Apparatus and method for operating a personal grooming appliance or household cleaning appliance

ABSTRACT

A system and method for treating a surface of a user’s body part, including: obtaining target surface condition information from a user’s body part surface using one or more condition sensors taken from a group consisting of: an optical sensor and a bio-sensor; classifying the target surface condition information using a machine learning classifier to determine an initial target surface condition classification; obtaining treatment progress information using a combination of motion sensor data and surface condition information from the one or more condition sensors; and classifying the treatment progress information using a machine learning classifier to determine a progress classification for treating the initial target surface condition classification.

BACKGROUND

There is a need for an ability for “smart” grooming or householdappliances and associated systems to leverage the use of combinations ofinternal sensor data and image data to improve the user’s experiencerelated to the grooming or household appliance.

SUMMARY

A first aspect of the current disclosure provide a system and method foroperating a personal grooming/household appliance that includes:providing a personal grooming/household appliance including (a) apowered and electronically controlled grooming/cleaning implement, and(b) at least one physical sensor taken from a group consisting of: anorientation sensor, an acceleration sensor, an inertial sensor, a globalpositioning sensor, a pressure sensor, a load sensor, audio sensor,magnetic sensor, humidity sensor, and a temperature sensor; providing acamera associated with the personal grooming/household appliance;deriving an augmented classification using one or more classifiersclassifying the physical sensor data and the image data; and modifyingoperation of the grooming/household appliance or implement based uponthe augmented classification.

In a detailed embodiment, the camera is located on the personalgrooming/household appliance.

Alternately, or in addition, the personal grooming/household appliancefurther includes a computer network interface transmitting and receivingdata over a computer network and the camera is located on a computerizeddevice that includes a computer network interface at least transmittingimage data over the computer network. In a further detailed embodiment,the operation modifying step is further based upon a treatment planimplemented by a software application operating, at least in part, onthe computerized device. In a further detailed embodiment, the treatmentplan is customized for a user of the grooming appliance. Alternatively,or in addition, the augmented classification is reconciled against thetreatment plan to determine treatment progress with respect to thetreatment plan. Alternatively, or in addition, the augmentedclassification is used, at least in part, to establish the treatmentplan.

In another detailed embodiment of the first aspect, the step of derivingthe augmented classification is performed by a single classifier.Alternatively, the method includes classifying sensor data received fromthe physical sensor using a trained machine learning classifier togenerate a physical classification; and classifying image data receivedfrom the camera using a trained machine learning classifier to generatean image classification; where the step of deriving the augmentedclassification is based upon the combination of the physicalclassification and the image classification.

In another detailed embodiment of the first aspect, the appliance is adental appliance; the grooming implement is a brush, a fluid nozzleand/or a flossing tape; and the augmented classification pertains to theposition of the grooming implement with respect to a user’s mouth. In afurther detailed embodiment, the modifying operation step deactivatesthe grooming implement when the augmented classification indicates thatthe grooming implement is outside of the user’s mouth. Alternatively, orin addition, the grooming implement is a motorized brush and themodifying operation adjusts a brush speed setting based upon theposition of the grooming implement with respect to the user’s mouth asindicated, at least in part, by the augmented classification.

In another detailed embodiment of the first aspect, the augmentedclassification pertains, at least in part, to whether or not thegrooming/cleaning implement is being implemented, and the step ofmodifying operation updates a maintenance setting based upon an amountof time that the grooming/cleaning implement is being implemented.

In another detailed embodiment of the first aspect, the augmentedclassification pertains, at least in part, to the position of thegrooming implement with respect to a user’s body part, and the step ofmodifying operation modifies operation of the grooming implement, basedat least in part, upon the position of the of the grooming implementwith respect to the user’s body part as indicated, at least in part, bythe augmented classification. In a further detailed embodiment, thegrooming implement is a dental appliance and the grooming implement is amotorized brush; and the step of modifying operation adjusts a speedsetting of the motorized brush based upon the position of the groomingimplement with respect to the user’s mouth as indicated, at least inpart, by the augmented classification. Alternatively, the groomingimplement is a shaving appliance and the grooming implement is amotorized shaving head; and the step of modifying operation adjusts aspeed setting of the shaving head based upon the position of thegrooming implement with respect to the user’s face as indicated, atleast in part, by the augmented classification. Alternatively, thegrooming implement is a shaving appliance and the grooming implement isa shaving cartridge; and the step of modifying operation adjusts anangle-of-attack setting of the shaving cartridge based upon the positionof the grooming implement with respect to the user’s face as indicated,at least in part, by the augmented classification. Alternatively, thestep of modifying operation adjusts a pressure sensitivity setting ofthe grooming implement based upon the position of the grooming implementwith respect to the user’s body part as indicated, at least in part, bythe augmented classification.

Alternatively, the augmented classification further includes a surfacecondition of the user’s body part; and the step of modifying operationadjusts a performance setting of the grooming implement based upon asurface condition at the position of the grooming implement with respectto the user’s face as indicated, at least in part, by the augmentedclassification. In a further detailed embodiment, the grooming applianceis a dental appliance and the surface condition pertains, at least inpart, to presence of plaque on a user’s teeth. Alternatively, thegrooming appliance is a shaving appliance and the surface conditionpertains, at least in part, to presence of whiskers on a user’s face.

In another detailed embodiment of the first aspect, the augmentedclassification pertains, at least in part, to the position of thecleaning implement with respect to a household target surface, and thestep of modifying operation modifies operation of the cleaningimplement, based at least in part, upon the position of the of thecleaning implement with respect to the household target surface asindicated, at least in part, by the augmented classification.

In another detailed embodiment of the first aspect, the augmentedclassification detects an identity of the user of the grooming/householdappliance and the modifying operation step applies an operation settingcustomized for the identified user.

It is a second aspect of the current disclosure to provide a system andmethod for operating a personal grooming appliance, including: providinga personal grooming/household appliance including at least one physicalsensor taken from a group consisting of: an orientation sensor, anacceleration sensor, an inertial sensor, a global positioning sensor, apressure sensor, and a load sensor, audio sensor, humidity sensor, and atemperature sensor; providing a camera associated with the personalgrooming/household appliance; classifying data received from thephysical sensor and from the camera using at least one trained machinelearning classifier to generate an augmented classification; andproviding user feedback information based upon the augmentedclassification; wherein the augmented classification pertains to acombination of a first state pertaining to a position of thegrooming/household appliance with respect to a user’s body-part orhousehold target surface and also pertains to a second state that isdifferent than the first state. In an embodiment, the second statepertains to an identity of a user. In an alternate embodiment, thesecond state pertains to an identity of the grooming appliance.

In an embodiment, the second state pertains to a surface condition of auser’s body part. In a further detailed embodiment, the user’s body partis a user’s teeth and the surface condition pertains to the presence ofplaque on the patient’s teeth. Alternatively, the surface condition isthe presence of whiskers or stubble on a user’s body part.Alternatively, or in addition, the first state also pertains to adirection of movement of the personal grooming appliance. In a furtherdetailed embodiment, the second state is an image classification derivedfrom image data from the camera. In yet a further detailed embodiment,the image classification pertains to, at least in part, an identity of ashaving lubricant being used. Alternatively, or in addition, the secondstate is a stroke pressure classification derived from the physicalsensor.

In an embodiment, the second state is an image classification derivedfrom image data from the camera. In a further detailed embodiment, theimage classification pertains to an emotion of a user of the groomingappliance. In yet a further detailed embodiment, the imageclassification pertains to a negative emotion of the user of thegrooming appliance, and the feedback information provides advice forimproving the user’s experience with the grooming appliance.

In an embodiment, the image classification pertains to at least one of apre-treatment or post-treatment condition. In a further detailedembodiment, the image classification pertains to a pre-treatmentcondition and the feedback information provides treatment instructionsbased upon the combination of pre-treatment condition and a position ofthe grooming/household appliance.

In an embodiment, the image classification pertains to an identity of anobject used with grooming/household-cleaning along with thegrooming/household appliance. In a further detailed embodiment, thefeedback information includes marketing information (e.g., coupons,promotions, advertisements and the like) related to the object.

In an embodiment, the feedback information includes marketinginformation related to the image classification. In a detailedembodiment, the image classification pertains to a condition of a user’sskin and the feedback information includes a recommended product fortreating the skin condition. In a further detailed embodiment, thefeedback information also includes a product application technique usingthe grooming appliance.

In an embodiment, the image classification pertains to a condition of auser’s body part and the feedback information includes a recommendationfor a product to apply to the body part along with a product applicationtechnique using the grooming appliance.

In an embodiment, the second state pertains to motion of thegrooming/household appliance. Alternatively, or in addition, the step ofgenerating the augmented classification is performed by a singleclassifier.

It is a third aspect of the current disclosure to provide a system andmethod for operating a personal grooming appliance, including: providinga personal grooming appliance including at least one motion sensor takenfrom a group consisting of: an orientation sensor, an accelerationsensor, and an inertial sensor; providing a camera associated with thepersonal grooming appliance; classifying data received from the motionsensor and from the camera using at least one trained machine learningclassifier to generate an augmented classification; and providing userfeedback information based upon the augmented classification; whereinthe augmented classification pertains to a combination of a first statepertaining to a position of the grooming appliance with respect to auser’s body part and also pertains to a second state that is differentthan the first classification. In a further detailed embodiment, thesecond state pertains to an identity of a user. Alternatively, thesecond state pertains to an identity of the grooming appliance.Alternatively, or in addition, the second state pertains to a surfacecondition of a user’s body part. Alternatively, or in addition, thefirst state also pertains to a direction of movement of the personalgrooming appliance.

In an embodiment the second state is an image classification derivedfrom image data from the camera. In a further detailed embodiment, theimage classification pertains to an emotion of a user of the groomingappliance. Alternatively, the image classification pertains to at leastone of a pre-treatment or post-treatment condition. In a furtherdetailed embodiment, the image classification pertains to apre-treatment condition and the feedback information provides treatmentinstructions based upon the combination of pre-treatment condition and aposition of the grooming appliance.

It is a fourth aspect of the current disclosure to provide a systemand/or method for operating a personal grooming appliance, comprising:providing at least one of a camera associated with the personal groomingappliance or a bio-sensor associated with the personal groomingappliance; providing a personal grooming appliance having at least onemotion sensor such as an orientation sensor, an acceleration sensor,and/or an inertial sensor; classifying at least one of image datareceived from the camera or bio-sensor data received from the bio-sensorto classify a surface condition of a surface of a user’s anatomy using afirst learning network classifier to generate an initial surfacecondition classification; generating user treatment information basedupon the initial surface condition classification and communicating theuser treatment information to the user; classifying motion data receivedfrom the motion sensor to classify motion of the personal groomingappliance with respect to the surface of the user’s anatomy using asecond learning network classifier to generate at least one of arelational motion classification or a relational positionclassification; generating user treatment progress information basedupon a subsequent surface condition classification and based upon the atleast one relational motion classification or relational positionclassification; and communicating the user treatment progressinformation to the user.

In an embodiment, the personal grooming appliance further includes acomputer network interface transmitting and receiving data over acomputer network; and the camera is located on a computerized devicethat includes a computer network interface at least transmitting imagedata over the computer network. In a further detailed embodiment, themethod further includes a step of modifying operation of the groomingappliance based upon the user treatment progress information.Alternatively, or in addition, the step of generating user treatmentinformation based upon the surface condition classification includesgenerating a treatment plan based, at least in part, on the surfacecondition information. In a further detailed embodiment, the treatmentplan is implemented by a software application operating, at least inpart, on the computerized device. In a further detailed embodiment, thetreatment plan is customized for a user of the grooming appliance.Alternatively, or in addition, the user treatment progress informationis reconciled against the treatment plan to determine treatment progresswith respect to the treatment plan. Alternatively, or in addition, themethod further includes a step of modifying the treatment plan basedupon, at least in part, user treatment progress information. In anembodiment, the modifying step follows a step of determining that theinitial surface condition classification is not correct. In anembodiment, the method further includes the step of communicating themodified treatment plan to the user.

It is a fifth aspect of the current disclosure to provide a systemand/or method for operating a personal grooming appliance, including:providing a computerized device including a camera and a networkinterface that transmits image data from the camera over a computernetwork; providing a personal grooming appliance including, (a) anorientation sensor, an acceleration sensor, an inertial sensor, apressure sensor, and/or a load sensor, and (b) a computer networkinterface transmitting and receiving data over the computer network;providing a software application operating, at least in part, on thecomputerized device; classifying image data received from the camera togenerate an image classification using one or more learning networkclassifiers; generating a treatment plan based, at least in part, on theimage classification; customizing the treatment plan based upon userinformation accessible to the software application; implementing atleast a portion of the customized treatment plan by the softwareapplication; classifying sensor data received from the at least onesensor to classify use of the personal grooming appliance with respectto the surface of the user’s anatomy, using one or more learning networkclassifiers, to generate a relational grooming appliance useclassification; and generating user treatment plan progress informationbased upon the relational grooming appliance use classification andcommunicating the user treatment plan progress information to the user.

In a further detailed embodiment, the step of classifying image dataincludes identifying the user’s anatomy. Alternatively, or in addition,the step of generating the relational grooming appliance useclassification is based upon classifying a combination of the image dataand the sensor data. Alternatively, or in addition, the method furtherincludes a step of modifying operation of the grooming appliance basedupon the relational grooming appliance use classification.Alternatively, or in addition, the user treatment progress informationis reconciled against the treatment plan to determine treatment progresswith respect to the treatment plan. Alternatively, or in addition, theuser information accessible to the software application includes userprofile information collected by the software application.Alternatively, or in addition, the user information accessible to thesoftware application includes information derived from the relationalgrooming appliance use classification. Alternatively, or in addition,the method further includes the step of training the one or morelearning network classifiers based upon how the user operates thegrooming appliance. Alternatively, or in addition, the method furtherincludes the step of training the one or more learning networkclassifiers based upon user information collected by the softwareapplication, where the user information may be collected by the softwareapplication is based, at least in part, upon the user’s interactionswith the software application.

It is a sixth aspect of the current disclosure to provide a method fortreating a surface of a user’s body part, including: obtaining targetsurface condition information from a user’s body part surface using oneor more condition sensors such an optical sensor and/or a bio-sensor;classifying the target surface condition information using a machinelearning classifier to determine an initial target surface conditionclassification; obtaining treatment progress information using acombination of motion sensor data and surface condition information fromthe one or more condition sensors; andclassifying the treatment progressinformation using a machine learning classifier to determine a progressclassification for treating the initial target surface conditionclassification.

In a more detailed embodiment to the sixth aspect, the one or morecondition sensors is provided on at least one of an examinationinstrument or a grooming appliance. Alternatively, or in addition, themethod further includes displaying a representation of the treatmentprogress information. In an embodiment, he the displayed representationis a time-lapse representation; or, in another embodiment, the displayedrepresentation is a real-time representation.

In a more detailed embodiment to the sixth aspect, the method includesmodifying a setting of a treatment system based upon, at least in part,the treatment progress classification. In a further detailed embodiment,the modifying step occurs substantially in real time while the treatmentsystem is treating the user’s body part surface. In a further detailedembodiment, the one or more condition sensors is provided a treatmentinstrument of the treatment system. In yet a further detailedembodiment, the treatment instrument is an oral care instrument.

In a more detailed embodiment to the sixth aspect the method includesmodifying settings of a treatment system based upon the target surfacecondition classification. Alternatively, or in addition, the methodfurther includes evaluating change of the user’s body part surfacecondition over time based upon successive target surface conditionclassifications. Alternatively, or in addition, the progressclassification indicates that the initial target surface conditionclassification is incorrect. In such a case, the method may furtherinclude, generating an initial treatment plan based upon the initialtreatment classification and modifying the initial treatment plan upondetermining that the initial target surface condition classification isincorrect.

These and other aspects and objects of the current disclosure willbecome apparent by the following description, the appended claims andthe attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

In the following, embodiments of the present disclosure are described inmore detail with reference to the figures, in which

FIG. 1 shows a schematic block diagram of an apparatus according to anembodiment of the current disclosure,

FIG. 2 shows an example of a target surface to be treated with themovable treatment device,

FIG. 3 shows a further example of a target surface to be treated withthe movable treatment device,

FIG. 4 shows a schematic block diagram of a recurrent neural networkthat may be used in embodiments disclosed herein,

FIG. 5 shows a schematic block diagram of a GRU neural network that maybe used in embodiments disclosed herein,

FIG. 6A shows a schematic block diagram of an LSTM neural network thatmay be used in embodiments disclosed herein,

FIG. 6B shows a schematic block diagram of an LSTM neural network withone layer at different time instances,

FIG. 7 shows a schematic block diagram of an LSTM neural network withtwo layers at different time instances,

FIG. 8 shows a block diagram of a method according to an embodiment ofthe current disclosure,

FIG. 9 shows a schematic block diagram of an apparatus according to anembodiment of the current disclosure,

FIG. 10 shows a schematic block diagram of an apparatus according to afurther embodiment of the current disclosure,

FIG. 11 shows a schematic block diagram of an apparatus according to afurther embodiment of the current disclosure,

FIG. 12 shows a block diagram of a method according to an embodiment ofthe current disclosure,

FIG. 13 a block diagram representation of a networked system accordingto embodiments of the current disclosure,

FIG. 14 is a flow diagram representation of a decision tree according toan embodiment of the current disclosure,

FIG. 15 is a block diagram representation of a networked systemutilizing an augmented or hybrid machine learning classificationaccording to an embodiment of the current disclosure, and

FIG. 16 is a block diagram representation of an alternate networkedsystem utilizing an augmented or hybrid machine learning classificationaccording to an embodiment of the current disclosure.

DETAILED DESCRIPTION

Equal or equivalent elements or elements with equal or equivalentfunctionality may denoted in the following description by equal orequivalent reference numerals. However, similar or equivalent elementsmay also be represented by different reference numerals based upon theembodiment.

In the following, reference will be made to personal grooming appliancesand/or household appliances as non-limiting examples of “movabletreatment devices.” For example, personal grooming appliances, mayinclude shaving appliances (manual shavers, electric shavers, trimmers,epilators, chemistry based hair removal and the like), dental appliances(manual toothbrushes, electric toothbrushes, polishers, waterjets,ultrasound appliances, flossing appliances and the like), scrubbers(exfoliating brushes and the like), cosmetic applicators, and hairstyling appliances (hair brushes, trimmers/cutters, hair dryers,straighteners, curlers and the like). Such grooming appliances may alsohave certain medical and/or dental examination/diagnosis/treatment usesand will be described herein. In these examples, the surface to betreated is a portion of a user’s anatomy, such a user’s teeth, face,legs, etc. Also, household appliance may include surface cleaners,polishers, pressure-washers, floor cleaners, vacuum cleaners, windowcleaners and the like. In these examples, the surface to be treated maybe a household surface such as a floor, a wall, a counter-top, a sink, awindow, a mirror, a vehicle surface, and the like.

Furthermore, an order of any method steps of a method may only bedescribed as a non-limiting example. Accordingly, unless expresslystated as being performed in a specific order, any method steps asdescribed herein may also be executed in any other order than described.

Although some aspects will be described in the context of an apparatusor device, it will be understood that these aspects also represent adescription of the corresponding method, where a block or devicecorresponds to a method step or a feature of a method step. Analogously,aspects described in the context of a method or method step alsorepresent a description of a corresponding block or item or feature of acorresponding apparatus or device.

FIG. 1 shows an apparatus or system 10 according to an exemplaryembodiment. A movable treatment device 11 is depicted. The movabletreatment device 11 may comprise an inertial sensor 13. Furthermore, themovable treatment device 11 may be configured to treat a target surface12. As will be described below, a camera 22 may also be associated withthe device 11 (as will be described below a camera is “associated” inthat the camera may be included as part of the treatment device 11 orthe camera may be apart from the treatment device, such as in a handheldsmartphone or in a smart display or mirror - essentially, any networkedor wirelessly connected camera).

As can be seen, the movable treatment device 11 may be located at acertain position relative to a target surface 12, for example in, at, onor next to the target surface 12. The target surface 12 itself may bedivided into one or more zones 21 ₁, 21 ₂, 21 ₃, ..., 21 _(n). Themovable treatment device 11 may be moved or be located at a positionrelative to at least one of said zones 21 ₁, 21 ₂, 21 ₃, ..., 21 _(n).

The apparatus 10, as depicted in FIG. 1 , may be configured to perform alocalization of the movable treatment device 11 relative to the targetsurface 12.

The apparatus 10 may comprise a motion pattern recognition device 14.The motion pattern recognition device 14 may be configured todiscriminate between two or more movement classifications such as motionpatterns 15 ₁, 15 ₂, 15 ₃, ..., 15 _(n) that are contained in a set 15of motion patterns of the movable treatment device 11. In other words,the movable treatment device 11 may be moved, e.g. by a user using saidmovable treatment device 11, in different linear and/or rotationaldirections. Accordingly, each motion of the movable treatment device 11may represent a respective or individual motion pattern. The motionpattern recognition device 14 may comprise a set 15 of different motionpatterns. The set 15 of motion patterns may comprise two or more of saidaforementioned respective or individual motion patterns 15 ₁, 15 ₂, 15₃, ..., 15 _(n). The motion pattern recognition device 14 may beconfigured to discriminate between these two or more motion patterns 15₁, 15 ₂, 15 ₃, ..., 15 n. That is, the motion pattern recognition device14 may be configured to distinguish a first motion pattern 15 ₁ from asecond motion pattern 15 ₂.

The movement of the movable treatment device 11 may be detected by atleast one inertial sensor 13. The inertial sensor 13 is a sensor basedon inertia and may comprise at least one of an accelerometer, agyroscope and a magnetometer. The inertial sensor 13 may provide sensordata representing at least one of a linear velocity, an angularvelocity, a linear acceleration, an angular acceleration and a g-force.The inertial sensor 13 may be part of an inertial measurement unitcomprising one or more inertial sensors.

The apparatus 10 may comprise an interface 16 for receiving at least oneinertial sensor data 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) from the inertialsensor 13 and for providing the at least one inertial sensor data 17 ₁,17 ₂, 17 ₃, ..., 17 _(n) to the motion pattern recognition device 14.The at least one inertial sensor data 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n)represents a movement of the movable treatment device 11. In otherwords, when the movable treatment device 11 moves, the inertial sensor13 senses this motion and creates at least one inertial sensor data 17₁, 17 ₂, 17 ₃, ..., 17 _(n). Accordingly, the at least one inertialsensor data 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) represents the respectivemotion of the moved treatment device 11.

The motion pattern recognition device 14 may comprise a neural network18. The neural network 18 may be a deep learning network. The neuralnetwork 18 may be configured to receive the at least one inertial sensordata 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) and to map the at least one inertialsensor data 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) to at least one of the motionpatterns 15 ₁, 15 ₂, 15 ₃, ..., 15 n contained in the set 15 of motionpatterns. This mapping is indicated in FIG. 1 by means of the dashed andsolid arrows 19 ₁, 19 ₂, 19 ₃, 19 ₄. The arrow 19 ₃ that is drawn insolid lines may exemplarily indicate that the neural network 18successfully mapped the at least one inertial sensor data 17 ₁, 17 ₂, 17₃, ..., 17 _(n) to the third motion pattern 15 ₃.

The different motion patterns 15 ₁, 15 ₂, 15 ₃, ..., 15 n that arecontained in the set 15 are exemplarily symbolized by differentgeometrical shapes (circle, rectangle, triangle, star) merely forillustration purposes. The motion patterns of the movable treatmentdevice 11 are, of course, not limited to these specific geometricalshapes.

According to the inventive principle, the motion patterns 15 ₁, 15 ₂, 15₃, ..., 15 n are each associated with one or more different zones 21 ₁,21 ₂, 21 ₃, ..., 21 _(n) of the target surface 12. This is indicated bymeans of the dashed and solid arrows 20 ₁, 20 ₂, 20 ₃, ..., 20 _(n). Ascan be seen, the first motion pattern 15 ₁ may be associated with afirst zone 21 ₁ of the target surface 12, as is indicated by the dashedarrow 20 ₁. The second motion pattern 15 ₂ may be associated with asecond zone 21 ₂ of the target surface 12, as is indicated by the dashedarrow 20 ₂. The third motion pattern 15 ₃ may be associated with a thirdzone 21 ₃ of the target surface 12, as is indicated by the arrow 20 ₃that is drawn in solid lines. The fourth motion pattern 15 ₄ may beassociated with a fourth zone 21 ₄ of the target surface 12, as isindicated by the dashed arrow 20 ₄.

The arrow 20 ₃ that is drawn in solid lines may exemplarily indicatethat the third motion pattern 15 ₃, to which the at least one inertialsensor data 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) was successfully mapped bythe neural network 18, is associated with the third zone 21 ₃ of thetarget surface 12.

Accordingly, the mapping of the at least one inertial sensor data 17 ₁,17 ₂, 17 ₃, ..., 17 _(n) with the at least one motion pattern 15 ₁, 15₂, 15 ₃, ..., 15 n indicates an estimation of the location of themovable treatment device 11 with respect to the one or more zones 21 ₁,21 ₂, 21 ₃, ..., 21 _(n) of the target surface 12. In the presentexample, the mapping of the at least one inertial sensor data 17 ₁, 17₂, 17 ₃, ..., 17 _(n) with the third motion pattern 15 ₃ indicates anestimation of the location of the movable treatment device 11 withrespect to the third zone 21 ₃ of the target surface 12.

In other words, the neural network 18 successfully mapped the receivedat least one inertial sensor data 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) to thethird motion pattern 15 ₃. Since, according to this example, the thirdmotion pattern 15 ₃ is associated with the third zone 213, the apparatus10 retrieves the information that the movable treatment device 11 islocated at the third zone 21 ₃, or that the movable treatment device 11at least was located at the third zone 21 ₃ at the time when the atleast one inertial sensor data 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) wascreated.

Thus, the apparatus 10 may be configured to localize the movabletreatment device 11 relative to the target surface 12 simply by means ofthe executed motion, or motion pattern, of the movable treatment device11.

According to an embodiment, the movable treatment device 11 may be apersonal grooming appliance and the target surface 12 may be a bodyportion to be treated by the movable treatment device 11.

For example, the movable treatment device 11 may be a razor or a groomerfor shaving or grooming a body portion of a user’s body. The user’s body(or a part of the body) may be the target surface 12 in this case. Theuser’s body 12 may be separated into different zones, for instance, aleft cheek zone, a right cheek zone, a chin zone and so on. By executinga predetermined motion pattern with the razor 11 the apparatus 10 maylocalize the razor 11 relative to the user’s body. For instance, if therazor 11 executes a motion pattern that is directed into an upper leftcorner with the razor 11 being tilted to the left, the apparatus 10 maylocalize the razor 11 as being located in the left cheek zone, forexample. Accordingly, the apparatus 10 may localize the razor 11 at theuser’s face simply by its executed motion pattern.

As a further example, the movable treatment device 11 may be a householdappliance and the target surface 12 may be a surface of a floor, a wall,a furniture or the like. For example, the movable treatment device 11may be a vacuum cleaner and the target surface 12 may be the floor of aroom. The room 12 may be separated into different zones, for instance, aleft top corner of the room, a right bottom corner of the room, a centerof the room, underneath a bed located inside the room, and so on. Byexecuting a predetermined motion pattern with the vacuum cleaner 11 theapparatus 10 may localize the vacuum cleaner 11 relative to the floor ofthe room. For instance, if the vacuum cleaner 11 executes a motionpattern that is merely directed forwards and backwards with the lance ofthe vacuum cleaner 11 being lowered near to the ground, the apparatus 10may localize the vacuum cleaner 11 as being located in the “underneaththe bed” zone, for example. Accordingly, the apparatus 10 may localizethe vacuum cleaner 11 inside the room simply by its executed motionpattern.

According to a further embodiment, the movable treatment device 11 maybe an oral care device and the target surface 12 may be a dentition,wherein the dentition 12 is separated into different dental zones 21 ₁,21 ₂, 21 ₃, ..., 21 _(n), wherein the mapping of the at least oneinertial sensor data 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) with the at leastone motion pattern 15 ₁, 15 ₂, 15 ₃, ..., 15 n indicates an estimationof the location of the oral care device 11 with respect to the one ormore dental zones 21 ₁, 21 ₂, 21 ₃, ..., 21 _(n) of the dentition 12.

The oral care device may be a toothbrush, in particular an electrictoothbrush. The oral care device may also be at least one of a dentalfloss, a polisher, a plaque removing device, an ultrasound device and awaterjet device. In some embodiments, the oral care device may also bean oral examination device.

According to this example, by executing a predetermined motion patternwith the oral care device 11 the apparatus 10 may localize the oral caredevice 11 relative to the dentition. For instance, if the oral caredevice 11 executes a motion pattern that is merely directed upwards anddownwards with the oral care device 11 being tilted to the left, theapparatus 10 may localize the oral care device 11 as being located in aleft upper dental zone of the upper jaw, for example. Accordingly, theapparatus 10 may localize the oral care device 11 relative to the user’sdentition simply by its executed motion pattern.

According to an embodiment, the dentition may be separated into ninedental zones, wherein a first dental zone corresponds to the left buccalside of the upper and lower jaw of the dentition, a second dental zonecorresponds to the occlusal side of the left and right side of the upperjaw of the dentition, a third zone corresponds to the occlusal side ofthe left and right side of the lower jaw of the dentition, a fourthdental zone corresponds to the left lingual side of the upper and lowerjaw of the dentition, a fifth dental zone corresponds to the rightbuccal side of the upper and lower jaw of the dentition, a sixth dentalzone corresponds to the right lingual side of the upper and lower jaw ofthe dentition, a seventh dental zone corresponds to the labial side ofthe upper and lower jaw of the dentition, an eighth dental zonecorresponds to the palatal side of the upper jaw of the dentition, aninth dental zone corresponds to the oral side of the front lower jaw ofthe dentition.

According to a further embodiment, at least one predetermined motionpattern 15NB that may be additionally contained in the set 15 of motionpatterns may be associated with a zone 21NB outside the target surface12, or not related to the target surface 12, wherein the mapping of theat least one inertial sensor data 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) withthe at least one predetermined motion pattern 15NB indicates that themovable treatment device 11 is located in said zone 21NB that is outsidethe target surface 12, or not related to the target surface 12.

In other words, the zone 21NB outside the target surface 12 may be azone that is not directly related to the target surface 12. For example,if the movable treatment device 11 may be a toothbrush, then said zone21NB outside the target surface 12 may be a zone outside the dentition.Accordingly, this zone 21NB may indicate that the user is not brushinghis teeth. Thus, this zone may also be referred to as a zone ‘NotBrushing’, abbreviated by ‘NB’. This zone 21NB may be the at least onezone of the target surface 12, or this zone 21NB may be an additionalzone in addition to the one or more zones 21 ₁, 21 ₂, 21 ₃, ..., 21 _(n)of the target surface. However, this particular zone 21NB outside thetarget surface 12 is not limited to the above described example of teethbrushing.

FIG. 2 shows a dentition 12 for illustrating the above describedexample. The dentition 12 may be the target surface. The dentition 12may be separated into nine dental zones 1 a to 9 a. Optionally, a tenthzone NB may exist. This tenth zone NB is a zone outside the dentition12. Thus, this tenth zone NB is not explicitly illustrated in FIG. 2 .Since this tenth zone NB is not related to one of the dental zones ofthe dentition 12, and therefore not concerned with brushing the teeth ofthe dentition 12, this tenth zone NB may also be referred to as a ‘NotBrushing’ zone.

As can be seen in FIG. 2 , a first dental zone 1 a may correspond to theleft buccal side of the upper and lower jaw of the dentition 12. Asecond dental zone 2 a may correspond to the occlusal side of the leftand right side of the upper jaw of the dentition 12. A third zone 3 amay correspond to the occlusal side of the left and right side of thelower jaw of the dentition 12. A fourth dental zone 4 a may correspondto the left lingual side of the upper and lower jaw of the dentition 12.A fifth dental zone 5 a may correspond to the right buccal side of theupper and lower jaw of the dentition 12. A sixth dental zone 6 a maycorrespond to the right lingual side of the upper and lower jaw of thedentition 12. A seventh dental zone 7 a may correspond to the labialside of the upper and lower jaw of the dentition 12. An eighth dentalzone 8 a may correspond to the palatal side of the upper jaw of thedentition 12. A ninth dental zone 9 a may correspond to the oral side ofthe front lower jaw of the dentition 12.

FIG. 3 shows a dentition 12 for illustrating a further example. Thedentition 12 may be the target surface. The dentition 12 may beseparated into sixteen dental zones 1 b to 16 b. Optionally, aseventeenth zone NB may exist. This seventeenth zone NB is a zoneoutside the dentition 12. Thus, this seventeenth zone NB is notexplicitly illustrated in FIG. 3 . Since this seventeenth zone NB is notrelated to one of the dental zones of the dentition 12, and thereforenot concerned with brushing the teeth of the dentition 12, thisseventeenth zone NB may also be referred to as a ‘Not Brushing’ zone.

As can be seen in FIG. 3 , a first dental zone 1 b may correspond to theleft buccal side of the upper jaw of the dentition 12. A second dentalzone 2 b may correspond to the occlusal side of the left side of theupper jaw of the dentition 12. A third dental zone 3 b may correspond tothe occlusal side of the left side of the lower jaw of the dentition 12.A fourth dental zone 4 b may correspond to the left lingual side of theupper and lower jaw of the dentition 12. A fifth dental zone 5 b maycorrespond to the right buccal side of the upper and lower jaw of thedentition 12. A sixth dental zone 6 b may correspond to the occlusalside of the right side of the upper jaw of the dentition 12. A seventhdental zone 7 b may correspond to the occlusal side of the right side ofthe lower jaw of the dentition 12. An eighth dental zone 8 b maycorrespond to the palatal side of the upper jaw of the dentition 12. Aninth dental zone 9 b may correspond to labial side of the upper jaw ofthe dentition 12. A tenth dental zone 10 b may correspond to the labialside of the lower jaw of the dentition 12. An eleventh dental zone 11 bmay correspond to the palatal side of the upper jaw of the dentition 12.A twelfth dental zone 12 b may correspond to the oral side of the frontlower jaw of the dentition 12. A thirteenth dental zone 13 b maycorrespond to the left buccal side of the lower jaw of the dentition 12.A fourteenth dental zone 14 b may correspond to the left lingual side ofthe lower jaw of the dentition 12. A fifteenth dental zone 15 b maycorrespond to the right buccal side of the lower jaw of the dentition12. A sixteenth dental zone 16 b may correspond to the right lingualside of the lower jaw of the dentition 12.

FIGS. 2 and 3 have only been described as non-limiting examples. Thetarget surface 12 may also comprise more or less than the exemplarilydescribed nine or sixteen dental zones. Furthermore, thetenth/seventeenth dental zone NB outside the target surface 12 isoptional. The exact distribution of the one or more dental zones of thedentition 12 may vary from the examples described above.

The apparatus 10 may be self-learning as regards the localization of themovable treatment device relative to the target surface 12. Theapparatus 10 may make use of artificial intelligence, for instance, byexploiting deep learning networks. For example, the apparatus 10 maymake use of classifiers developed by artificial intelligence and/orlearning networks; and further, the apparatus 10 (and the systemsdescribed herein) may further teach such classifiers. Accordingly, theapparatus 10 for performing the localization of the movable treatmentdevice 11 relative to the target surface 12 may enhance its performanceover time by using the neural network 18.

According to an embodiment, the neural network 18 may be a RecurrentNeural Network (RNN).

For example, the neural network may be a Long Short Term Memory (LSTM)network or a Gated Recurrent Unit (GRU) network.

RNNs may suffer from the so-called vanishing gradient problem, whereingradients vanish quickly with more number of layers. Vanishing gradientsmay lead to rather slow training rates. Thus, LSTM networks and/or GRUnetworks may be used to avoid the vanishing gradient problem.

An LSTM network is an artificial neural network containing LSTM blocksin addition to regular network units. An LSTM block contains gates thatdetermine when the input is significant enough to remember, when itshould continue to remember or when it should forget the value, and whenit should output the value.

FIG. 4 shows an example for a RNN in its most general form. A neuralunit 40 may be fed with an input 41 at a certain time instant t. Theinput 41 may be a single value or a vector comprising two or morevalues. The input 41 at the certain time instant t may also besymbolized with Xt.

The neural unit 40 may optionally also comprise a further input 42. Thisfurther input 42 may be provided from a neural unit (not depicted here)at a previous time instant t-1.

The neural unit 40 may comprise at least one gate 43, which may providea mathematical operation. In this example, the gate 43 is a single tanhgate.

The neural unit 40 may comprise at least one output 46. The output 46may comprise the result of the operation of the tanh gate 43 that hasbeen fed with the input 41 and optionally the further input 42. Theoutput 46 may lead to a hidden state 45, which will be explained later.

The neural unit 40 may optionally comprise a further output branch 46which branches off from the above mentioned output result of theoperation of the tanh gate 43 fed with the input 41 and optionally thefurther input 42.

In FIG. 4 , each depicted line may carry an entire vector, from theoutput of one node to the inputs of others. Lines merging, for instanceat 47, denote concatenation, while a line forking, for instance at 48,denote its content being copied and the copies going to differentlocations. This holds true also for the other neural networks that willbe described in the following with reference to the following Figures.

FIG. 5 shows an example of a GRU network. The GRU network comprises aneural unit 50. In addition to the above described RNN neural unit 40,the GRU neural unit 50 may comprise two further gates, namely a firstsigmoid gate 53 and a second sigmoid gate 54. Furthermore, the GRUneural unit 50 may comprise pointwise operations 55, 56, 57, 58, 59,like vector addition 58, for example.

FIG. 6A shows an example of an LSTM network that may be exploited as theneural network 18 in the apparatus 10. The LSTM may comprise a neuralunit 60 which may, in the case of LSTM networks, also be referred to asan LSTM block. In addition to the above described neural units 40, 50,the neural unit 60 of the depicted LSTM network may comprise a cellstate, which is the horizontal line 61 running through the top of theneural unit 60. The neural unit 60 may receive a cell state input 62 andmay create a cell state output 66.

The neural unit 60 may further comprise four gates 43, 53, 54, 63. Forexample, it may comprise a further sigmoid gate 63 compared to the GRUnetwork described above. Information may be removed or added to the cellstate (horizontal line 61) by means of these gates 43, 53, 54, 63.

FIG. 6B shows a further example in which previous and subsequent states(with respect to the time instant t) of the neural unit are depicted. Inparticular, a neural unit 60 t at a time instant t is depicted.Furthermore, a further neural unit 60 t-1 at a previous time instant t-1is depicted. Still further a further neural unit 60 t+1 at a subsequenttime instant t+1 is depicted. The depicted neural units 60 t-1, 60 t, 60t+1 may represent the same neural unit but at different points in time,namely at the time instant t, at a previous time instant t-1 and at asubsequent time instant t+1.

The above described input 41, also symbolized by the letter X, maycomprise the at least one sensor data 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n)from the inertial sensor 13. The input X may be time dependent, thusX=X(t). In particular, the depicted input Xt may comprise a sensor data17 ₂ acquired during the considered time instant t, the depicted inputXt-1 may comprise a sensor data 17 ₁ acquired during a previous timeinstant t-1, and the depicted input Xt+1 may comprise a sensor data 17 ₃acquired during a subsequent time instant t+1.

As can further be seen in FIG. 6B, the neural unit 60 t-1, 60 t, 60 t+1may, in each depicted time instant t-1, t, t+1, provide, for instance byprediction, a respective output value yt-1, yt, yt+1. The output valuey(t) may be a single value or a vector comprising one or more vectorelements.

The output value y(t) may be calculated as: yt = softmax (Why • ht+b)

The output value y(t) may, for instance, comprise probabilistic values,as will be explained in more detail with respect to FIG. 7 . Forexample, the output value y(t) may be a vector comprising one or morevector elements, wherein each vector element may represent one of themotion patterns 15 ₁, 15 ₂, 15 ₃, ..., 15 n, or in more detail whereineach vector element may represent a probabilistic value indicating howprobable it is that the input X(t), i.e. the inertial sensor data 17 ₁,17 ₂, 17 ₃, ..., 17 _(n), may correspond to one of the motion patterns15 ₁, 15 ₂, 15 ₃, ..., 15 _(n).

Furthermore, the depicted neural units 60 t-1, 60 t, 60 t+1 may bearranged in the same layer, namely in a first layer. Some examples maycomprise one or more further layers, wherein each layer may comprise itsown neural unit(s). Such examples may be described later with referenceto FIG. 7 for example. However, examples and embodiments with at least afirst layer will be described with further reference to FIG. 6B.

According to this embodiment, the neural network 18 may comprise a firstlayer, wherein said first layer comprises a neural unit 60 t, wherein ata first time instant t the at least one inertial sensor data Xt 17 ₂ isinput into the neural unit 60 t of the first layer. At a subsequentsecond time instant t+1 a second inertial sensor data Xt+1 173 and atleast one output ht 46 of the neural unit 60 t of the previous firsttime instant t are input into the neural unit 60 t+1 of the first layer.

FIG. 7 shows a further example, wherein the neural network 18 maycomprise at least two layers, namely a first layer 71 and a second layer72. The first layer 71 comprises at least a first neural unit 60 t, andthe second layer 72 comprises at least a second neural unit 70 t.

As can be seen, the sensor data 17 ₁, 17 ₂, 17 ₃ that is acquired duringdifferent time instances t-1, t, t+1 may be fed as input Xt-1, Xt, Xt+1into the respective neural unit 60 t-1, 60 t, 60 t+1 of the first layer71.

The output 46 t-1, 46 t, 46 t+1 of each neural unit 60 t-1, 60 t, 60 t+1of the first layer 71 may be fed as an input into the respective neuralunits 70 t-1, 70 t, 70 t+1 of the second layer 72.

The neural units 60 t-1, 60 t, 60 t+1 of the first layer 71 and theneural units 70 t-1, 70 t, 70 t+1 of the second layer 72 may beidentical. Alternatively, the internal structure of the neural units 60t-1, 60 t, 60 t+1 of the first layer 71 and the neural units 70 t-1, 70t, 70 t+1 of the second layer 72 may differ from each other.

According to the embodiment as shown in FIG. 7 , the neural network 18may comprise at least a first layer 71 and a second layer 72, whereinthe first layer 71 may comprise a first neural unit 60 t and wherein thesecond layer 72 may comprise a second neural unit 70 t, wherein at afirst time instant t the at least one inertial sensor data Xt 17 ₂ isinput into the first neural unit 60 t of the first layer 71, and whereinan output ht 46 of the first neural unit 60 t is input into the neuralunit 70 t of the second layer 72.

So far a signal path in a vertical direction, i.e. from a bottom firstlayer 71 to a top second layer 72 has been described. However, in theembodiment of FIG. 7 also a signal path in a horizontal direction isshown.

As can be seen, the cell state output Ct 66 of a first neural unit 60 tat a first time instant t and/or the output ht 46 of the first neuralunit 60 t at the first time instant t may be fed as an input into thefirst neural unit 60 again, namely into the first neural unit 60 t+1 ata subsequent time instant t+1. As already mentioned above, the neuralunit 60 itself may be the same neural unit but it may only be depictedin the Figures as a plurality of concatenated neural units 60 t-1, 60 t,60 t+1 for ease of illustration of the states of the neural unit 60 atthe different time instances t-1, t, t+1. In other words, the horizontalsignal path may describe the signal path of the neural unit 60 atdifferent subsequent time instances t-1, t, t+1. The same holds true forthe second layer 72 and any further layers.

Accordingly, the depicted subsequent time instances t-1, t, t+1 mayrepresent a length 77 during which the neural network 18 may sample andprocess the acquired sensor data 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n). Saidlength 77 may therefore be referred to as a run length, a sample lengthor a sample period. For example, the sample length 77 may correspond toone second, wherein the time instances t-1, 5 t, t+1 may be fractions ofsaid one second. For example a sample period 77 may have a length offifty samples, i.e. of fifty time instances. The neural network 18 mayrun once during a sample period, or the neural network 18 may runpermanently over two or more sample periods.

Thus, according to a further embodiment, the neural network 18 maycomprise at least a first layer 71 and a second layer 72, wherein thefirst layer 71 may comprise a first neural unit 60 t and wherein thesecond layer 72 may comprise a second neural unit 70 t, wherein at afirst time instant t the at least one inertial sensor data Xt 17 ₂ maybe input into the first neural unit 60 t of the first layer 71, andwherein at least one output ht 46 of the first neural unit 60 t may beinput into the neural unit 70 t of the second layer 72. So far it may bethe same as described above. However, additionally, at a subsequentsecond time instant t+1, a second inertial sensor data Xt+1 17 ₃ and atleast one output ht 46 of the first neural unit 60 t at the first timeinstant t is input into the first neural unit 60 t+1 at the subsequentsecond time instant t+l.

As mentioned above, several mathematical operations may be executed bythe neural network 18, in the gates 43, 53, 54, 63. In the example shownin FIG. 7 the following mathematical operations may be executed at thedifferent stages:

$\begin{array}{l}{{\overline{h}}_{j} = {\sum\limits_{h \in C{(i)}}{h_{h},}}} \\{i_{j} = \sigma\left( {W^{(i)}x_{j} + U^{(i)}{\overline{h}}_{j} + b^{(i)}} \right),} \\{f_{jh} = \sigma\left( {W^{(f)}x_{j} + U^{(f)}h_{x} + b^{(f)}} \right),} \\{o_{j} = \sigma\left( {W^{(o)}x_{j} + U^{(o)}{\overline{h}}_{j} + b^{(o)}} \right),} \\{u_{j} = \text{tanh}\left( {W^{(u)}x_{j} + U^{(o)}{\overline{h}}_{j} + b^{(o)}} \right),} \\{c_{j} = i_{j} \odot u_{j} + {\sum\limits_{k \in C{(j)}}{f_{jk} \odot c_{k +}}}} \\{h_{j} = o_{j} \odot \text{tanh}\left( c_{j} \right),}\end{array}$

wherein

-   i(t) is the input gate’s activation vector-   f(t) is the forget gate’s activation vector-   o(t) is the output gate’s activation vector-   c(t) is the cell state vector-   h(t) is the output vector of an LSTM block or neural unit 60, 70

According to this example, the input sensor data Xt 172 may be anelement vector Xt ∈ ℝ⁶. For example it may be an input tensor Xt ∈ ℝ⁶,[Ax, Ay, Az, Gx, Gy, Gz]^(T) Furthermore weights W(t) and bias valuesb(t) are depicted in FIG. 7 , wherein in this example:

-   Weights Why ∈ ℝ^(12×256)-   Bias bt ∈ ℝ¹²

Furthermore, the output vector y(t) may be calculated as:

yt = softmax(Why ⋅ ht + b)

The depicted hidden states h(t) may also be element vectors, for exampleelement vectors comprising 256 elements ht ∈ ℝ²⁵⁶.

Furthermore, the depicted hidden states C(t) may also be elementvectors, for example element vectors comprising 256 elements Ct ∈ ℝ²⁵⁶.

As mentioned above, the input inertial sensor data Xt 17 ₂ may be anelement vector Xt ∈ ℝ⁶ comprising six vector elements, for example aninput tensor Xt ∈ ℝ⁶, [Ax, Ay, Az, Gx, Gy, Gz]^(T). These vectorelements [Ax, Ay, Az, Gx, Gy, Gz]^(T) may also be referred to asinertial sensor data portions.

According to an embodiment, the at least one inertial sensor data 17 ₁may comprise at least three inertial sensor data portions of the groupcomprising a linear velocity in x, y and z direction, an angularvelocity with respect to the x, y and z axes, a linear acceleration inx, y and z direction, and an angular acceleration with respect to the x,y and z axes.

In other words, the inertial sensor 13 may provide inertial sensor data17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) at the one or more time instances t-1, t,t+1, wherein the inertial sensor data 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) maydepend on the current orientation and motion of the movable treatmentdevice 11 at one observed time instance t-1, t, t+1. Each of theinertial sensor data 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) may be a vectorcomprising at least three, or in other examples at least six vectorelements, wherein said vector elements represent the above mentionedinertial sensor data portions, wherein at least one of said inertialsensor data portions may be zero.

Accordingly, the inertial sensor data 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n)(vectors), and in particular the sensor data portions (vector elements),may represent the current motion pattern of the movable treatment device11 as sampled during a sample period 77 comprising one or moresubsequent time instances t-1, t, t+1.

According to an embodiment as depicted in FIG. 7 , the at least oneinertial sensor data 17 ₂ (vector) may comprise one or more inertialsensor data portions (vector elements), wherein an input to the neuralunit 60 t at a first time instant t is a respective inertial sensor data17 ₂ comprising the one or more inertial sensor data portions retrievedduring said first time instant t. At least one inertial sensor data 17₁, 17 ₂, 17 ₃, ..., 17 _(n) may be sampled during a sample time 77.

The neural network 18 may map the at least one sampled inertial sensordata 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) which has been sampled during thesample time 77 to at least one motion pattern 15 ₁, 15 ₂, 15 ₃, ..., 15_(n) contained in the set 15 of motion patterns, as it was initiallydescribed with reference to FIG. 1 . After mapping, the selected onemotion pattern may be referred to as a mapped motion pattern.

In other words, the neural network 18 may receive the inertial sensordata 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) as an input x(t) and it may outputone or more probability values as an outputy(t). As mentioned above, inthe example shown in FIG. 7 , the output value y(t) may also be anelement vector comprising for example at least three, or at least six,or at least twelve vector elements. Each vector element of the outputvector y(t) may represent a probabilistic value for a motion pattern 15₁, 15 ₂, 15 ₃, ..., 15 _(n) that may be associated with a class or azone 21 ₁, 21 ₂, 21 ₃, ..., 21 _(n). In some embodiments, the outputvalue y(t) may be an element vector comprising, for example, at leasttwo to as many needed classes or zones, for example nine zones, twelvezones or sixteen zones.

Accordingly, the output vector y(t) may represent the different zones 21₁, 21 ₂, 21 ₃, ..., 21 _(n) of the target surface 12. For example, ifthe target surface 12 may comprise twelve zones (e.g. eleven dentalzones and a twelfth zone ‘NB’ for not brushing) then the element outputvector y(t) may comprise twelve vector elements, such as shown in theexample of FIG. 7 , wherein y(t) ∈ R¹².Accordingly, each vector elementmay represent one of the different zones 21 ₁, 21 ₂, 21 ₃, ..., 21 _(n)of the target surface 12.

As mentioned before, the vector elements may represent probabilityvalues. These probability values may represent the probabilistic valuefor each of the different zones 21 ₁, 21 ₂, 21 ₃, ..., 21 _(n) of thetarget surface 12. In other words, the neural network 18 may receive theat least one inertial sensor data 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) and mapthe at least one inertial sensor data 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) toat least one motion pattern 15 ₁, 15 ₂, 15 ₃, ..., 15 _(n), and sincesaid motion patterns 15 ₁, 15 ₂, 15 ₃, ..., 15 _(n) may each beassociated with one or more different zones 21 ₁, 21 ₂, 21 ₃, ..., 21_(n) of the target surface 12, the probability values may indicate howprobable it is, that the acquired at least one inertial sensor data 17₁, 17 ₂, 17 ₃, ..., 17 _(n) may correspond to one of the different zones21 ₁, 21 ₂, 21 ₃, ..., 21 _(n) of the target surface 12. This is calledthe mapping of the at least one inertial sensor data 17 ₁, 17 ₂, 17 ₃,..., 17 _(n) to at least one of the motion patterns 15 ₁, 15 ₂, 15 ₃,..., 15 _(n).

Since each motion pattern 15 ₁, 15 ₂, 15 ₃, ..., 15 _(n) may beassociated with one or more different zones 21 ₁, 21 ₂, 21 ₃, ..., 21_(n) of the target surface 12, the mapping of the at least one inertialsensor data 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) with the at least one motionpattern 15 ₁, 15 ₂, 15 ₃, ..., 15 _(n) indicates an estimation of thelocation of the movable treatment device 11 with respect to the one ormore zones 21 ₁, 21 ₂, 21 ₃, ..., 21 _(n) of the target surface 12. Thelocation of the treatment device 11 may be estimated because theinventive location detection may be based on the above mentionedprobability values in contrast to absolute valued geodata from a GPS,for instance.

In other words, the apparatus 10 may derive, from the neural network 18,an estimation in which zone 21 ₁, 21 ₂, 21 ₃, ..., 21 _(n) of the targetsurface 12 the movable treatment device 11 is located by simplyreceiving sensor data 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) and mapping saidsensor data 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) to motion patterns 15 ₁, 15₂, 15 ₃, ..., 15 _(n) being associated with one or more zones 21 ₁, 21₂, 21 ₃, ..., 21 _(n) of the target surface 12.

Thus, according to an embodiment, an output y(t) of the neural network18 may comprise one or more probability values for the estimation of thelocation of the movable treatment device 11 with respect to the one ormore zones 21 ₁, 21 ₂, 21 ₃, ..., 21 _(n) of the target surface 12.

According to yet a further embodiment, the motion pattern recognitiondevice 14 may be configured to determine from the at least one inertialsensor data 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) a mutual movement of themovable treatment device 11 and the target surface 12, and to remove thedetermined movement of the target surface 12 from the determinedmovement of the movable treatment device 11.

For example the movable treatment device 11 may be a toothbrush and thetarget surface 12 may be a user’s dentition. The user may turn his headwhile brushing his teeth. In this case the inertial sensor 13 wouldsense the mutual movement of the user’s head and the toothbrush, sincethe toothbrush is moved together with the head. This may lead to a wrongmotion detection, therefore to a wrong mapping, and finally to a wronglocalization based on the mapping.

However, according to the above embodiment the sensed or determinedmovement of the user’s head (target surface) 12 may be removed from thesensed mutual movement of the head and the toothbrush. In result onlythe desired movement of the toothbrush (treatment device) 11 remains. Aswill be described in further detail below, this movement of the user’shead (target surface) 12 may be detected by a camera associated with theapparatus 10, where image and/or video output of the camera may beclassified by neural network 18 (or by a separate learning network).

FIG. 8 shows a block diagram of an example of a method for performing alocalization of a movable treatment device 11 relative to a targetsurface 12, wherein the movable treatment device 11 comprises aninertial sensor 13 and wherein the movable treatment device 11 isconfigured to treat the target surface 12.

In block 801 the method comprises a step of discriminating between twoor more motion patterns 15 ₁, 15 ₂, 15 ₃, ..., 15 _(n) contained in aset 15 of motion patterns of the movable treatment device 11.

In block 802 the method comprises a step of receiving at least oneinertial sensor data 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) from the inertialsensor 13, the at least one inertial sensor data 17 ₁, 17 ₂, 17 ₃, ...,17 _(n) representing a movement of the movable treatment device 11.

In block 803 the method comprises a step of receiving and processing bymeans of a neural network 18 the at least one inertial sensor data 17 ₁,17 ₂, 17 ₃, ..., 17 _(n) and mapping/classifying the at least oneinertial sensor data 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) to at least onemotion pattern 15 ₁, 15 ₂, 15 ₃, ..., 15 _(n) contained in the set 15 ofmotion patterns, wherein said motion patterns 15 ₁, 15 ₂, 15 ₃, ..., 15_(n) contained in the set 15 of motion patterns are each associated withone or more different zones 21 ₁, 21 ₂, 21 ₃, ..., 21 _(n) of the targetsurface 12 so that the mapping/classifying of the at least one inertialsensor data 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) with the at least one motionpattern 15 ₁, 15 ₂, 15 ₃, ..., 15 _(n) indicates an estimation of thelocation of the movable treatment device 11 with respect to the one ormore zones 21 ₁, 21 ₂, 21 ₃, ..., 21 _(n) of the target surface 12.

FIG. 9 shows another exemplary apparatus 100 according to the currentdisclosure. The apparatus 100 may be similar to the above describedapparatus 10. Furthermore, all of the features described above withrespect to the apparatus 10 are combinable with the below describedapparatus 100, and vice versa.

The apparatus 100 may vary from the apparatus 10 (c.f. FIG. 1 ) in thatthe motion patterns 15 ₁, 15 ₂, 15 ₃, ..., 15_(n)may bemapped/classified to one or more class members 101A, 101B, ..., 104A,104B of different classes 101, 102, 103, 104 instead of different zones21 ₁, 21 ₂, 21 ₃, ..., 21 _(n) of a target surface 12.

Accordingly, the apparatus 100 is configured for classifying a motion ofa movable personal appliance 11, also referred to herein as a moveabletreatment device 11, comprising an inertial sensor 13. The apparatus 100comprises a motion pattern recognition device 14 configured todiscriminate between two or more motion patterns 151, 152, 153, ..., 15n contained in a set 15 of motion patterns of the movable personalappliance 11.

Furthermore, the apparatus 100 comprises an interface 16 for providingat least one inertial sensor data 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) fromthe inertial sensor 13 to the motion pattern recognition device 14,wherein the at least one inertial sensor data 17 ₁, 17 ₂, 17 ₃, ..., 17_(n) represents a motion of the movable personal appliance 11.

The motion pattern recognition device 14 comprises a neural network 18that is configured to receive the at least one inertial sensor data 17₁, 17 ₂, 17 ₃, ..., 17 _(n) and to map/classify the at least oneinertial sensor data 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) to at least onemotion pattern 15 ₁, 15 ₂, 15 ₃, ..., 15 _(n) contained in the set 15 ofmotion patterns, wherein the at least one mapped motion pattern 15 ₁, 15₂, 15 ₃, ..., 15 _(n) is associated with at least one class member 101A,101B, 102A, 102B, 103A, 103B, 104A, 104B of one or more classes 101,102, 103, 104 so that the at least one class member 101A, 101B,...,104A, 104B is selected based on the motion of the movable personalappliance 11.

In other words, the neural network 18 may map/classify the at least oneinertial sensor data 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) to at least onemotion pattern 15 ₁, 15 ₂, 15 ₃, ..., 15 _(n), e.g. in a way aspreviously described above with reference to FIGS. 1 to 8 . Since themapped motion patterns 15 ₁, 15 ₂, 15 ₃, ..., 15 _(n) may each beassociated with at least one class member 101A, 101B, ..., 104A, 104B ofone or more classes 101, 102, 103, 104, the at least one class member101A, 101B, ..., 104A, 104B may be selected based on the at least onemapped motion pattern 15 ₁, 15 ₂, 15 ₃, ..., 15 _(n) of the movablepersonal appliance 11, i.e. based on the motion of the movable personalappliance 11.

The non-limiting example of FIG. 9 shows four classes 101, 102, 103,104, wherein each class comprises two class members 101A, 101B, ..., nA,nB. However, there may be at least one class and each class may compriseat least two class members. There may also be more than two classes oreven more than the exemplarily depicted four classes.

As can be seen in the example of FIG. 9 , a first mapped/classifiedmotion pattern 151 may be associated with a class member 101A of thefirst class 101. An n^(th) mapped/classified motion pattern 154 may beassociated with a class member nB of the fourth class 104. A secondmapped/classified motion pattern 152 may be associated with two classmembers of different classes, for example with a class member 101B ofthe first class 101 and with a class member 102A of the second class102. A third mapped/classified motion pattern 153 may be associated withtwo class members of the same class, for example with two class members103A, 103B of the third class.

Generally at least one mapped/classified motion pattern 15 ₁, 15 ₂, 15₃, ..., 15_(n)may be associated with at least one class member 101A,101B, 102A, 102B, 103A, 103B, 104A, 104B of one or more classes 101,102, 103, 104.

In the following, some examples of classes and class members will bedescribed.

According to an embodiment, at least one class 101 of the one or moreclasses 101, 102, 103, 104 may comprise at least one class member 101A,wherein said one class 101 may represent a user group, and wherein saidat least one class member 101A may represent at least one user of saiduser group, wherein the at least one mapped motion pattern 15 ₁, 15 ₂,15 ₃, ..., 15 _(n) may be associated with the at least one class member101A for identifying said at least one user based on the motion of themovable personal appliance 11.

In other words, one of the classes 101, 102, 103, 104 may represent auser group, i.e. a group of users using the movable personal appliance11. The respective class may comprise at least one class member that mayrepresent one particular user of said user group. For example, the firstclass 101 may represent a user group, wherein said user group may be asingle household. In this example, the user group 101 may only containone class member 101A, i.e. one person. The inventive apparatus 100 maybe configured to identify said at least one user 101A simply based onthe motion of the movable personal appliance 11. Thus, the inventiveapparatus 100 may personalize any actions or interactions with said oneidentified user 101A, as will be described with some examples later.

According to a further embodiment, at least one class 101 of the one ormore classes 101, 102, 103, 104 may comprise at least two class members101A, 101B, wherein said one class 101 may represent a user group, andwherein said at least two class members 101A, 101B may represent atleast two users of said user group, wherein the at least one mappedmotion pattern 15 ₁, 15 ₂, 15 ₃, ..., 15 _(n) may be associated with oneof said at least two class members 101A, 101B for identifying at leastone user within the user group based on the motion of the movablepersonal appliance 11.

In other words, one of the classes 101, 102, 103, 104 may represent auser group, i.e. a group of users using the movable personal appliance11. The respective class may comprise at least one class member that mayrepresent one particular user of said user group. For example, the firstclass 101 may represent a user group, wherein said user group may be afamily. The classes 101A, 101B of said class 101 may represent thefamily members. For example, the user group 101 may comprise one or morefamily members, wherein a first class member 101A may represent themother of the family and a second class member 101B may represent achild of the family, for example.

The apparatus 100 may be configured to identify at least one user simplybased on the motion of the movable personal appliance 11. This may beachieved if every user may use the 1 movable personal appliance 11 in adifferent or individual way.

For example, in an embodiment the movable personal appliance 11 may be amovable oral care device, such as a toothbrush, in particular anelectric toothbrush. The movable oral care device may also be at leastone of a dental floss, a plaque removing device, an ultrasound deviceand a waterjet device.

To take up the example above, the mother 101A may use the toothbrush 11in a different way than the child 101B. The inertial sensor 13 of thetoothbrush 11 may provide its inertial sensor data 17 ₁, 17 ₂, 17 ₃,..., 17 _(n) to the motion pattern recognition device 14 comprising theneural network 18. The neural network 18 may map/classify the inertialsensor data 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) to at least one motionpattern 15 ₁, 15 ₂, 15 ₃, ..., 15 _(n).

For example, as shown in FIG. 9 , the mother may have a brushing stylethat corresponds to the first motion pattern 15 ₁. This motion pattern15 ₁ may be associated with class member 101A that represents themother. The child instead may have a different brushing style than themother, for example a brushing style that corresponds to the secondmotion pattern 15 ₂. This motion pattern 15 ₂ may be associated withclass member 101B that represents the child.

Thus, the apparatus 100 may identify a user of a user group simply basedon the motion of the movable personal appliance 11. As mentioned above,the inventive apparatus 100 may personalize any action or interactionwith the identified user.

According to an embodiment, the motion pattern recognition device 14 maybe configured to select, based on the step of identifying said at leastone user 101A, a user-specific motion pattern preset 115 comprising twoor more user-specific motion patterns 115 ₁, 115 ₂, 115 ₃, ..., 115 _(n)of the movable personal appliance 11 which are characteristic for saididentified at least one user 101A.

Such an example is shown in FIG. 10 . This embodiment may also bereferred to as a two-step process. In a first step 121, a user isidentified. The identified user may have a user specific motion patternpreset 115 that has been individually trained by the neural network 18.In a second step 122 the neural network 18 uses the user specific motionpatterns 115 ₁, 115 ₂, 115 ₃, ..., 115 _(n) from the user-specificmotion pattern preset 115. Thus, the inventive apparatus 100 may thenact and interact with each identified user individually.

In FIG. 10 a first step 121 is shown in which the neural network 18receives the at least one inertial sensor data 17 ₁, 17 ₂, 17 ₃, ..., 17_(n) and maps same to at least one of the motion patterns 15 ₁, 15 ₂, 15₃, ..., 15 _(n) that are contained in the set 15 of motion patterns. Theat least one mapped motion pattern, for example the nth notion pattern15 ₄, may be associated with a class member 101B of a first class 101.This procedure may correspond to the procedure as described above withreference to FIG. 9 .

The class 101 may be a user group and the class member 101B may be auser of said user group. To take up the above example, the identifieduser 101B may be the child of the family. The apparatus 100 may havestored user specific motion patterns. That is, the identified user, i.e.the child 101B, may have its own individual user specific preset 115 ofmotion patterns 115 ₁, 115 ₂, 115 ₃, ..., 115 _(n) stored in theapparatus 100. For any further actions following the identification inthe first step 121, the motion pattern recognition device 14, and inparticular the neural network 18, may use these user specific motionpatterns 115 ₁, 115 ₂, 115 ₃, ..., 115 _(n) belonging to the previouslyidentified user.

Thus, the neural network 18 may select, after the step 121 ofidentifying said at least one user 101B, at least one user-specificmotion pattern preset 115 comprising two or more user-specific motionpatterns 115 ₁, 115 ₂, 115 ₃, ..., 115 _(n) of the movable personalappliance 11 which are characteristic for said identified at least oneuser 101B.

Accordingly, in a second step 122 following the first step 121 ofidentifying the user, the neural network 18 may use the user specificpreset 115 of user specific motion patterns 115 ₁, 115 ₂, 115 ₃, ...,115 _(n) in replacement of the set 15 of motion patterns 15 ₁, 15 ₂, 15₃, ..., 15 _(n). That is, all of the herein described actions that canbe executed by the apparatuses 10, 100 by exploiting the set 15 ofmotion patterns 15 ₁, 15 ₂, 15 ₃, ..., 15 _(n) can also be executedindividualized or personalized for each identified user by theapparatuses 10, 100 by exploiting the user specific preset 115 of motionpatterns 115 ₁, 115 ₂, 115 ₃, ..., 115 _(n) instead of the set 15 ofmotion patterns 15 ₁, 15 ₂, 15 ₃, ..., 15 _(n).

Thus, according to an embodiment the neural network 18 may be configuredto replace, after the first step 121 of identifying said at least oneuser 101B, the set 15 of motion patterns by the selected user-specificmotion pattern preset 115, and to replace the two or more motionpatterns 15 ₁, 15 ₂, 15 ₃, ..., 15_(n)contained in the set 15 of motionpatterns by the two or more user-specific motion patterns 115 ₁, 115 ₂,115 ₃, ..., 115 _(n) contained in the user specific motion patternpreset 115.

Additionally or alternatively, the apparatus 100 may comprise at least asecond neural network. FIG. 11 shows such an example.

The example of the apparatus 100 of FIG. 11 may substantially correspondto the apparatus 100 of the example shown in FIG. 10 . The apparatus ofFIG. 11 differs from the apparatus of FIG. 10 in that the apparatus ofFIG. 11 may comprise a second neural network 182.

As can be seen in FIG. 11 , in a first step 121 a first neural network181 may execute the actions as described above, for example identifyinga user 101B of a user group 101. However, in a second step 122 theinertial sensor data 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) may be fed into saidsecond neural network 182. The second neural network 182 may use theuser specific preset 115 of motion patterns 115 ₁, 115 ₂, 115 ₃, ...,115 _(n) as described above.

In other words, after the first step 121 of identifying said at leastone user 101B, the motion pattern recognition device 14 may use thesecond neural network 182, wherein the second neural network 182 may beconfigured to receive the at least one inertial sensor data 17 ₁, 17 ₂,17 ₃, ..., 17 _(n) and to map the at least one inertial sensor data 17₁, 17 ₂, 17 ₃, ..., 17 _(n) to at least one user specific motion pattern115 ₁, 115 ₂, 115 ₃, ..., 115 _(n) contained in the user specific preset115 of motion patterns, wherein said user specific motion patterns 115₁, 115 ₂, 115 ₃, ..., 115 _(n) are each associated with at least oneclass member 102A, 102B of one or more classes 101, ..., 104 so that theat least one class member 102A, 102B is selected based on the motion ofthe movable personal appliance 11. In other words, the neural network 18may be a user specifically trained neural network.

Accordingly, the motion pattern recognition device 14 may be configuredto use the user specific preset 115 of user specific motion patterns 115₁, 115 ₂, 115 ₃, ..., 115 _(n) for user-specifically classifying themotion of the personal appliance 11 by means of the at least oneinertial sensor data 17 ₁, 17 ₂, 17 ₃, ..., 17 n.

As shown in the examples of FIGS. 10 and 11 , the apparatus 100 maycomprise at least one class 102 for classifying purposes in the secondstep 122. However, the apparatus 100 may comprise more than one class,as shown in the example of FIG. 9 , in the second step 122.

In said second step 122, for example after having identified aparticular user in the first step 121, different actions may beperformed by the personal appliance 11. For example, the personalappliance 11 may change its operation mode based on the identified user.For example, the personal appliance 11 may be electrical driven and itmay comprise a motor (See FIGS. 15 & 16 ), wherein the personalappliance 11 may change one or more motor specific characteristics, suchas frequency, amplitude or pulsation, based on the identified user.Additionally or alternatively, the personal appliance 11 may compriseone or more elements for communicating with or providing feedback to auser, for example a visual element, such as a light, e.g. a LED, or ahaptical element, such as a vibrational motor. For example, the personalappliance 11 may change a user experience based on the identified userby changing the operation mode of said elements for communicating, forinstance by changing LED lights to a different color or by providingdifferently pulsed feedback by the vibrational motor, based on theidentified user.

Additionally or alternatively to identifying a particular user of a usergroup, for example a family member of a family, the apparatus 100 may beconfigured to identify a particular user type. For example, if thepersonal appliance 11 was a toothbrush, some people start brushing theirteeth with their front teeth or incisors while some other people maystart brushing their teeth with their back teeth or molars. In a furtherexample, if the personal appliance was a razor, some people may shavewith the grain while some other people may shave against the grain.Summarizing a user type may be a type of user who uses the personalappliance 11 in a particular way. There may be two or more users thatcan be clustered into groups of user types. The previously describedexample of user identification instead identifies each userindividually.

According to an embodiment for identifying user types, at least oneclass 104 of the one or more classes 101, 102, 103, 104 may comprise atleast two class members nA, nB, wherein said one class 104 may representa user type of the movable personal appliance 11, wherein a first classmember nA may represent a first user type of the movable personalappliance 11 and wherein a second class member nB may represent a seconduser type of the movable personal appliance 11, wherein the at least onemapped motion pattern 15 ₁, 15 ₂, 15 ₃, ..., 15_(n)may be associatedwith either the first or the second class member nA, nB for identifyinga user type of the movable personal appliance 11 based on the motion ofthe movable personal appliance 11.

According to a further embodiment, the motion pattern recognition device14 may be configured to select, after the step of identifying said usertype, a user type specific motion pattern preset 115 comprising two ormore user type specific motion patterns 115 ₁, 115 ₂, 115 ₃, ..., 115_(n) of the movable personal appliance 11 which are characteristic forsaid identified user type, and wherein the neural network 18 may beconfigured to replace, after the step of identifying said user type, theset 15 of motion patterns by the selected user type specific motionpattern preset 115 and to replace the two or more motion patterns 151,152, 153, ..., 15 n contained in the set 15 of motion patterns by thetwo or more user type specific motion patterns 115 ₁, 115 ₂, 115 ₃, ...,115 _(n).

Everything that has been explained above with respect to the userspecific preset 115 of user specific motion patterns 115 ₁, 115 ₂, 115₃, ..., 115 _(n) also holds true for the user type specific preset 115of user type specific motion patterns 115 ₁, 115 ₂, 115 ₃, ..., 115_(n).

As mentioned above, the identified user types may be clustered into acluster or group of user types. Therefore, the apparatus 100 may performa cluster analysis in which a user may use the personal appliance 11 fora predetermined number of times before this user is clustered into aparticular user type group. For example, a user may use its razor fivetimes on five subsequent days. On four out of the five days the user mayshave against the grain. Thus, after the fifth day the apparatus 100 maycluster this user into a user type group in which all users shavingagainst the grain are clustered.

The cluster analysis may also be performed at shorter time intervals,i.e. switching the toothbrush 11 on and off may be done directlysuccessively. For example the user may switch on his electric toothbrush11 a first time, switch it off, and switch it on a second time torestart the toothbrush 11 again. At the time of restarting thetoothbrush 11, the inventive apparatus 100, and in particular the neuralnetwork 18, may also be restarted. When the toothbrush 11 is switchedon, it may collect information for the cluster analysis. However, atleast the neural network 18 shall restart every time before newinformation for the cluster analysis is collected. Summarizing, theapparatus 100 may repeatedly (e.g. five times) perform the clusteranalysis before finally clustering the user into a particular user typegroup.

After the user has been clustered into a particular user type specificgroup, the neural network 18 may use the associated user type specificpreset 115 of user type specific motion patterns 115 ₁, 115 ₂, 115 ₃,..., 115 _(n).

According to such an embodiment, the motion pattern recognition device14 may be configured to repeatedly perform a cluster analysis for apredetermined number of times, wherein in each said cluster analysis theneural network 18 may be configured to restart and to perform, after therestart, the step of receiving the at least one inertial sensor data 17₁, 17 ₂, 17 ₃, ..., 17 _(n) and to map the at least one inertial sensordata 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) to at least one motion pattern 15 ₁,15 ₂, 15 ₃, ..., 15 _(n) contained in the set 15 of motion patterns, andwherein the neural network 18 may be configured to select the user typespecific motion pattern preset 115 after performing the cluster analysisfor the predetermined number of times.

The inventive apparatus 100 may provide even more scenarios forclassifying a motion of the movable personal appliance 11. Therefore,reference shall be made to FIG. 9 again.

According to an embodiment, at least one class 102 of the one or moreclasses 101, 102, 103, 104 may comprise at least two class members 102A,102B, wherein said one class 102 may represent a handling evaluation ofthe movable personal appliance 11, wherein a first class member 102A mayrepresent a correct handling of the movable personal appliance 11 andwherein a second class member 102B may represent a wrong handling of themovable personal appliance 11, wherein the at least one mapped motionpattern 15 ₁, 15 ₂, 15 ₃, ..., 15 _(n) may be associated with either thefirst or the second class member 102A, 102B for evaluating the handlingof the movable personal appliance 11 based on the motion of the movablepersonal appliance 11.

In other words, the apparatus 100 may be configured to check whether auser of the movable personal appliance 11 may use the movable personalappliance 11 correctly or not. Of course, said one class 102representing the handling evaluation may also be used as a class in thesecond step 122 of the above described two-step procedures of FIGS. 10and 11 , e.g. after identifying a user and/or a user type.

According to a further embodiment at least one class 103 of the one ormore classes 101, 102, 103, 104 may comprise at least two class members103A, 103B, wherein said one class 103 may represent a quality of motionexecution of the movable personal appliance 11, wherein a first classmember 103A may represent a good motion execution of the movablepersonal appliance 11 and wherein a second class member 103B mayrepresent a bad motion execution of the movable personal appliance 11,wherein the at least one mapped motion pattern 151, 152, 153, ..., 15 nmay be associated with either the first or the second class member 103A,103B for evaluating a quality of motion execution of the movablepersonal appliance 11 based on the motion of the movable personalappliance 11.

In other words, the apparatus 100 may be configured to check whether auser of the movable personal appliance 11 may use the movable personalappliance 11 in a good way or in a bad way. A good way may be a way ofperforming the motion of the movable personal appliance as intended,while a bad way may be a way of performing the motion of the movablepersonal appliance 11 as not intended. For example, if the personalappliance 11 was a toothbrush, then the apparatus may check whether theuser may have a good or a bad brushing technique.

Of course, said one class 103 representing the quality of motionexecution may also be used as a class in the second step 122 of theabove described two-step procedures of FIGS. 10 and 11 , e.g. afteridentifying a user and/or a user type. Yet a further embodiment of theapparatus 100 may be similar to the apparatus 10 as described withreference to FIGS. 1 to 8 .

According to such an embodiment, at least one class 104 of the one ormore classes 101, 102, 103, 104 may comprise at least two class membersnA, nB, wherein said one class 104 may represent a location of themovable personal appliance 11 with respect to a target surface 12,wherein a first class member nA may represent a first location zone 211of the movable personal appliance 11 with respect to the target surface12 and wherein a second class member nB may represent a second locationzone 212 of the movable personal appliance 11 with respect to the targetsurface 12, wherein the at least one mapped motion pattern 151, 152,153, ..., 15 n may be associated with at least one of the first and thesecond class members nA, nB for localizing the movable personalappliance 11 within at least one of the first and the second locationzones 211, 212 based on the motion of the movable personal appliance 11.

In other words, the one class 104 may represent a target surface 12. Theclass members nA, nB of said one class 104 may represent different zones211, 212 of said target surface 12. Accordingly, the localization of themovable personal appliance 11 with respect to the target surface 12 maybe executed by the apparatus 10 in the same or at least a similarfashion as described above with respect to the apparatus 10 withreference to FIGS. 1 to 8 .

Of course, said one class 104 representing the location of the movablepersonal appliance 11 with respect to the target surface 12 may also beused as a class in the second step 122 of the above described two-stepprocedures of FIGS. 10 and 11 , e.g. after identifying a user and/or auser type.

According to an embodiment, at least one class 101 of the one or moreclasses 101, 102, 103, 104 may comprise at least one class member 101A,wherein said one class 101 may represent an appliance type (e.g, shavingappliance, dental appliance, broom), and wherein said at least one classmember 101A may represent at least one specific appliance in theappliance type group, wherein the at least one mapped motion pattern 15₁, 15 ₂, 15 ₃, ..., 15 _(n) may be associated with the at least oneclass member 101A for identifying said at least one specific appliancebased on the motion of the movable personal appliance 11.

The neural network 18 of the apparatus 100 may comprise the same orsimilar features as the neural network 18 of the apparatus 10 that hasbeen described with reference to FIGS. 4 to 7 . Thus, it shall bebriefly referred to FIG. 7 again.

According to an embodiment, the neural network 18 may comprise at leasta first and a second layer 71, 72, wherein each layer may comprise aneural unit 60, 70, wherein at a first time instant t the at least oneinertial sensor data Xt 17 ₂ may be input into the neural unit 60 of thefirst layer 71, and wherein at a subsequent second time instant t+1 asecond inertial sensor data Xt+1 17 ₃ and at least one output ht 46 ofthe previous first time instant t may be input into the neural unit 60of the first layer 71, and/or wherein at the subsequent second timeinstant t+1 the at least one output ht 46 of the first time instant tmay be input into the neural unit 71 of the second layer 72.

Everything that has been described above with respect to any features ofthe neural network 18 of the apparatus 10 as shown in FIGS. 4 to 7 alsoholds true for the neural network 18 of the apparatus 100 as describedwith reference to FIGS. 9 to 11 .

FIG. 12 shows a block diagram of an inventive method for classifying amotion of a movable personal appliance 11 that comprises an inertialsensor 13.

In block 1201 the method comprises a step of discriminating between twoor more motion patterns 15 ₁, 15 ₂, 15 ₃, ..., 15 _(n) contained in aset 15 of motion patterns of the movable personal appliance 11.

In block 1202 the method comprises a step of providing at least oneinertial sensor data 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) from the inertialsensor 13 to the motion pattern recognition device 14, the at least oneinertial sensor data 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) representing amotion of the movable personal appliance 11.

In Block 1203 the method comprises a step of receiving and processing,by means of a neural network 18, the at least one inertial sensor data17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) and mapping the at least one inertialsensor data 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) to at least one motionpattern 15 ₁, 15 ₂, 15 ₃, ..., 15 _(n) contained in the set 15 of motionpatterns, wherein the at least one mapped motion pattern 15 ₁, 15 ₂, 15₃, ..., 15 _(n) is associated with at least one class member 101A, 101B,102A, 102B, ..., nA, nB of at least one class 101, 102, 103, 104 so thatthe at least one class member 101A, 101B, 102A, 102B, ..., nA, nB isselected based on the motion of the movable personal appliance 11.

According to yet a further example of the apparatus 10, 100 the movabletreatment device 11 may be a personal grooming appliance and the targetsurface 12 may be a body portion to be treated by the movable treatmentdevice 11.

According to yet a further example of the inventive apparatus 10, 100the movable treatment device 11 or the movable personal appliance 11 maycomprise a pressure sensor for sensing a pressure applied onto a targetzone by the personal appliance and/or a load sensor for sensing a motorload of a motor that may drive the personal appliance.

Respective sensor data of the pressure sensor and/or the load sensor maybe fed as input into the neural unit 18, in addition or alternatively tothe at least one inertial sensor data 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n).

According to yet a further example of the inventive apparatus 10, theapparatus 10 may comprise an output interface for outputting to a userthe one or more zones 21 ₁, 21 ₂, 21 ₃, ..., 21 _(n) of the targetsurface 12 in which the movable treatment device 11 is located.

According to yet a further example of the inventive apparatus 100, theapparatus 100 may comprise an output interface for outputtinginformation to a user, said information being related to the one or moreclasses 101, 102, 103, 104 and/or to the one or more class members 101A,101B, ..., nA, nB of the one or more classes 101, 102, 103, 104.

In each of the herein described embodiments, sensor data 17 ₁, 17 ₂, 17₃, ..., 17 _(n) can be stored on the movable personal appliance ortreatment device 11 and later on can be fed into the apparatus 10, 100,in a way as described above. Any post processing of this stored sensordata 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) into different zones or classes maybe used to show a consumer or user on a dashboard how well and whatzones they covered, what they forgot, what was in target vs out oftarget. This data may be shown as one usage or aggregated uses over time(i.e. show the consumer or user a simple dashboard of how they have beenbrushing over the week).

The following features may also be included:

-   Attention mechanism (add on to the RNN)-   Prefiltering work-   Removing head position dependency (looking at linear accl)-   Dynamic time warping for user ID (finger print)-   Local high freq sampling and 8 bit FFT to diff lingual and buccal    (based on cheek damping of signal – this would be done by simple on    device classifier followed by raw signal + device classifier into    RNN-   Not only train a position predictor, but also train a “brushing    correctly vs not”-   Doing cluster analysis (have user brush 1-5 times before placing    them into a bucket) to put user in a defined space that uses a    custom trained RNN for that type of user

FIG. 13 discloses an example networked appliance system 1000 accordingto the current disclosure. The networked appliance system includes agrooming appliance 1003 which is illustrated in this example as a razorappliance. But the appliance may be any grooming appliance, householdappliance or treatment device 11 disclosed herein. In the currentexample, the razor appliance 1003 includes a grooming implement such asa removable razor cartridge 1006, a razor handle 1002, an internal powersource 1118, an optional multi-color LED display 1050, and an optionalcamera 1082.

As discussed above and herein, the razor appliance 1003 may include aplurality of internal sensors such as motion sensor(s), orientationsensor(s), cartridge ejection sensor(s), new cartridge detectionsensors, and/or pressure sensor(s) associated with the handle 1002and/or razor cartridge 1006. The shaving appliance 1003 may also includean appliance circuit 1052 connected to receive (via a data connection)sensor signals from the plurality of sensors contained within the razorappliance 1003. In the current embodiment, the network appliance system1000 also includes a base station 1301, where the base station includesa seat 1056 for receiving and engaging with the handle 1002 of the razorappliance 1003. In the current embodiment, the base station 1301 may bepowered by electricity via an electric cord 1058 that may be pluggedinto a standard electrical outlet. The seat 1056 may include electrodes(not shown) that are adapted to engage with and/or mate withcorresponding electrodes (again not shown) on the razor appliance handle1002. Through such electrodes the base station 1301 may provide power tocharge the power source (such as a rechargeable battery) 1118 in therazor appliance 1003 and/or may provide an electrical connection for thetransfer of data signals from the sensor circuit 1052 within the razorhandle 1002 to a base station circuit 1060 residing within the basestation 1301. It is also within the scope of the current disclosure thatpower may be provided from the base station 1052 to the razor’s powersource 1118 by a non-connected capacitive coupling as known in the art,or any other wireless mechanisms that are known forwirelessly/contact-less transferring power from a first power source toa rechargeable power source. It is also within the scope of the currentdisclosure that the power source 1118 may be removable, such asdisposable batteries and/or rechargeable batteries that are charged bysomething other than the base station 1301. Further, it is within thescope of the current disclosure that data transmitted/received betweenthe razor 1003 and the base station 1301 may be via wireless dataconnection, such as a Bluetooth connection and the like. It is alsowithin the scope of the current disclosure that some or all of themechanisms, circuitry and/or functionality of the base station 1301 asdescribed herein can reside within razor 1003. It will be appreciatedthat, while the base station 1301 is described in this example asassociated with the razor 1003, similar base stations and base stationfunctionalities may also be associated with other appliances disclosedherein.

In the current embodiment, the base station 1301 includes base stationcircuitry 1060 that includes processor(s) and corresponding circuitryfor receiving the sensor signals (and/or information derived from thesensor signals) and converting the sensor signals/information intoassociated analysis/mapping/classification information as describedherein. The base station circuitry 1060, in the current embodiment, alsoincludes a network circuitry for a wireless data communication (e.g.,such as a cellular and/or WiFi connection) with a computer network 1062such as a cellular network and/or an internet network. The base station1301 may also include a visual display 1064, such as an LCD displayand/or a similar text or image display device as known to those ofordinary skill, where such display device 1064 may be controlled by thebase station circuitry 1060. The base station 1301 may also include asound actuator 1066 also controlled by the base station circuitry 1060,where the sound actuator 1066 may include a speaker or similarsound-making component.

The networked shaving appliance system 1000 also includes a computerizedand networked user interface device 1080. The computerized and networkeduser interface device 1080 can be in the form of a smart phone, a tabletcomputer, personal assistant device, a laptop or desktop computer, smartdisplay, smart mirror, a computerized wearable appliance such as a smartwatch or smart glasses, and the like. The computerized and networkeduser interface device 1080 may include a display 1066, and a user inputdevice such as a cursor control device 1068 (or a touch screen or avoice activated control, or a motion sensor, or an eye movement sensorand the like as are readily available to the art), a camera 1070 andassociated processing circuitry 1072. The computerized and networkeduser interface device 1080 may operate to perform various softwareapplications such as a computerized tool which may be in the form of apersonal application 1073 (see FIGS. 15&16 ) associated with theappliance 11 as will be discussed in further detail herein. In thecurrent embodiment, the application 1073 is a personal shavingapplication and may include a graphical user interface 1074, which maybe displayed on the display screen 1066 and controlled and/or receiveuser input therein from the user input devices such as thecursor-controlled device 1068 and/or the touch screen. The user devicecircuitry 1072 may include a network circuit for connecting wirelesslywith the computer network 1062 for the purpose of receiving and/ortransmitting data over the computer network 1062.

As also illustrated in FIG. 13 , the computer network 1062 may havevarious computer servers and/or distributed computing devices(collectively labeled as 1076) also accessible thereto and mayadditionally include various data storage devices 1077 operativelycoupled by a data connection thereto. For example, the softwareapplication 1073 may include operations being performed on one or moreof the computer servers/devices 1076 and/or on the device circuitry1072. Likewise, data storage associated with the software application1073 may be within one or more of the data storage devices 1077 and/oron the device circuitry 1072.

At a very high level, one or more of the appliance circuit 1052, basestation circuit 1060, user device circuitry 1072 and/or processorsassociated with the distributed computing environment 1076 comprise asensor circuit for receiving the sensor signals from the razor appliance1003 and for analyzing/mapping/classifying the sensor signals asdescribed herein. Likewise, again at a very high level, one or more ofthe appliance circuit 1052, base station circuit 1060, user devicecircuitry 1072 and/or processors associated with the distributedcomputing environment 1076 comprise an image processing circuit forreceiving the image data from the camera 1082 and/or 1070 and foranalyzing/mapping/classifying the image data as described herein. Thisanalysis, mapping and/or classification information will also becommunicated over the computer network 1062 so that a computerized toolwhich may be in the form of the software application 1073 operating onthe networked user interface device 1080 may receive the analysis,mapping and/or classification information (or at least portions thereof)associated with a user of the computerized device 1080 from the network1062. The computerized tool in the form of the software application 1073may also be configured to receive user profile data information from theuser via the graphical user interface 1074 provided by the softwareapplication 1073. Further, the software application 1073 may process theanalysis, mapping and/or classification information received from thecomputer network 1062 with the user profile data provided by the userthrough the software application to generate user feedback informationassociated with the user’s experience with the appliance 11 (razor 1003in this example) as described herein; and then finally, communicate thatuser feedback information to the user via the graphical user interface1074 provided by the computerized tool as also described herein and/orvia the LED’s 1050 on the razor 1003, and/or via the visual display 1064on the base station, and/or via the sound actuator 1066 on the basestation.

As shown in FIG. 14 , specific examples of measurement information orshave event information for the razor 1003 may include (withoutlimitation) razor movement information 1102 based upon acceleration inX, Y and Z directions derived from sensor data received form a 3-axisaccelerometer 1110; razor orientation information 1104 based upon angleinformation derived from sensor signals received from a 3-axis gyrometer 1130; razor heading information 1106 based upon relationship withmagnetic north derived from sensor signals received from a 3-axismagnetometer 1210; cartridge pivot movement information 1108 (alsocartridge presence, cartridge contact and/or trimmer contact) based uponrelationship of a magnet with respect to a pivot plunger derived fromsensor signals received from a 3-axis magnetometer 1160; razor-in-handinformation (information corresponding to a user gripping the handle1002) 1110 based upon barometric pressure derived from sensor signalsreceived from the capacitive sensor 1420; and razor attitude information1112 derived from sensor signals received from the barometric pressuresensor 1440.

As also shown in FIG. 14 , razor attitude information 1114 can bederived from a combination of the razor movement information 1102, razororientation information 1104 and razor heading information 1106.Cartridge contact information 1116 can be derived from pivot movementinformation 1108. Stroke event information can be derived from acombination of the razor attitude information 1114, razor contactinformation 1116, razor-in-hand information 1110 and razor attitudeinformation 1112.

As further shown in FIG. 14 , the measurement and shave eventinformation may also include image information provided by cameras andassociated analysis, mapping and/or classification. For example, as willbe described in further detail below, hair growth direction information1120 may be provided by image information received through the camera1082/1070 and through a stubble analysis 1122 performed on the imageinformation using appropriate computer learning or statisticalanalyzers, mapper and/or classifiers described herein. Consequently,relative stroke direction information 1124 (which determines whether ornot the stroke directions are with or against the direction of hairgrowth on the user’s face) can be derived from a combination of razorattitude information 1114, stroke event information 1118 and the hairgrowth direction information 1120 provided by the image analysis.Similarly, over-stroke information or over-strokes with/against thegrain can be determined based upon a combination of sensor readingstaken from a plurality of the same sensors and image information as usedfor shave direction information and/or relative shave directioninformation.

Additional sensors, as discussed herein, may include thermistors forsensing handle operating temperature and/or in-handle temperature;capacitive sensors for sensing razor-in-hand; multi-capacitance sensorsfor sensing grip positions; clocks for sensing time; acoustic sensorsfor sensing shave performance (such as with or against grain) and thelike.

Another aspect to the current disclosure is that the shave eventinformation can be cumulative shave event information starting at a timewith the system senses or is informed that a new shaving cartridge 1006is attached to the razor 1003. New cartridge determination may beprovided by receiving sensor signals associated with a cartridge ejectbutton on the razor appliance 1003. Similarly, new cartridgedetermination information may be provided by having a new-cartridgesensor becoming active upon the cartridge ejections occurring (such as amechanical switch being set for activation when a cartridge is ejected),where the new-cartridge sensor may be then actuated when the newcartridge is inserted. New cartridge information may also be manuallyindicated by the user such as through the software application 1073 orby the user pressing a reset button (or the like), for example, on thebase station 1301. Additionally, new cartridge information may bedetected by the razor appliance 1003 by detecting a unique I.D. for eachrazor cartridge that is attached to the handle 1002. For example, aunique I.D. can be a barcode on the cartridge sensed by an associatedbarcode reader on the handle; can be an RFID tag on the cartridge sensedby an associated RFID reader on the handle; can be an I.D. on thecartridge communicated to the handle by magnetic, electric or capacitivedata communication; can be a physical I.D. such as a physical key on thecartridge 1006 that is sensed by the handle 1002; and so forth.Essentially, any known manner for the appliance 1003 or system 1000 todetect or be notified when a new razor cartridge 1006 is coupled to thehandle 1002 (the new cartridge event) will begin the collection pointfor cumulative shave event data where that cumulative shave event datawill be thereafter associated with the age of the new razor cartridge1006. This cumulative shave event information can be used to calculateor estimate, for example, the sharpness of the associated bladescontained within the cartridge 1006.

The systems and methods herein may include training one or moreconvolutional neural networks (“CNN”) for determining the method oftreating a target surface 12. The CNN can be used to identify targetsurfaces 12, treatment implements 11 and practitioner informationrelevant to the treatment method determined by the CNN. The CNN mayutilize training images and/or audio data to train the convolutionalneural network and may receive one or more training images or audiofiles to be utilized by the CNN to determine elements that definesurface types, treatment implements, and practitioners. Once the CNN istrained, a camera 22 may capture an image (e.g., a digital image) of thetarget surface, implement, practitioner, for analysis by the CNN. Thecamera 22 may then provide image data 23 for analysis, mapping and/orclassification as described herein. The analysis of the captured imagedata 23 may include a determination of the target surface 12 type,target surface 12 condition, target surface diagnosis, implement 11type, user information as well as treatment method, additional relevanttreatment products and treatment regimen information. The CNN and RNNstructures may be used in sequence or in parallel to evaluate the dataand determine a surface.

The image capture logic of the camera 22 and the software tool in theform of a computer application 1073 may include and/or utilize softwarecomponents, hardware circuitry, firmware, and/or other computinginfrastructure, as described herein. As described in more detail below,the image capture logic may facilitate capturing, storing,preprocessing, analyzing, transferring, and/or performing otherfunctions on a digital image data 23. The application 1073 may beconfigured for providing one or more user interfaces 1074 to the user,which may include questions, options, and the like.

Features detected by the analysis/mapping/classifying system may includeedges, shapes, colors, which may be used to identify age, gender,emotion, skin type, hair type, floor type, fabric type, tooth color,skin color, pimples/acne, redness, shine of skin & hair.Products/Devices -toothbrushes, comb, hair brush, ProX, razors, groomingdevices, Swiffer, beauty/cosmetic device. Accordingly, the remoteprocessing servers and/or computers 1076 include a memory component1077, which stores training logic and analyzing logic. The traininglogic may facilitate creation and/or training of the CNN, and thus mayfacilitate creation of and/or operation of the convolutional neuralnetwork. The analyzing logic may cause the processing servers and/orcomputers 1076 to receive data from the mobile computing device 1080 (orother computing device) and process the received data for providing atreatment product recommendation, etc through the user interface 1074.

A training computer or server 1076 may be coupled to the network 1062 tofacilitate training of the CNN. For example, a trainer may provide oneor more images to the CNN via the training computer or server 1076. Thetrainer may also provide information and other instructions to informthe CNN which assessments are correct and which assessments are notcorrect. Based on the input from the trainer, the CNN may automaticallyadapt, as described in more detail below.

It should also be understood that while the training computer or server1076 is described as performing the convolutional neural networkprocessing, this is merely an example. RNNs or multi-layer-perceptron(MLP) may be used as alternative network architectures that and appliedto video or other digital data including audio data. Any of thesenetworks can be used since they are capable of analyzing, mapping and/orclassifying video and/or sensor information. The convolutional neuralnetwork processing may be performed by any suitable computing device, asdesired.

The present system may include a convolutional neural network (“CNN”)that functions as a surface treatment expert system. For example, a CNNmay be stored as logic in the memory component of a computing device.The CNN may be configured to receive a training image (or a plurality oftraining images) and take raw image pixels from the training image asinput and automatically learn feature extractors that are relevant fordetermining surface, implement, and practitioner types from a captureddigital image. Recent advances in a machine learning technique calleddeep learning have resulted in breakthrough performance in the realm ofneural networks, such as is described in U.S. Pat. No. 8,582,807.Deep-learning-type neural networks utilize a plurality of layersinspired by the human visual cortex.

The CNN may be trained using predefined features and/or auto-learnedfeatures. After the CNN is trained, it may be used to determine surfacetreatment options from a captured image of the user from the learnedfeatures. In some instances, the CNN can learn to identify importantfeatures in an image through a process called supervised learning.Supervised learning generally means that the CNN is trained by analyzingexamples of images in which the surface treatment options have beenpre-defined. Depending on the accuracy that is desired, the number oftraining images may vary from a few images to a continuous input ofimages to provide for continuous training. Regardless, after training,the CNN learns key features for predicting the treatment methodaccurately for a wide range of surface types.

The CNN may include a plurality of stages. A first stage may includepreprocessing and a second stage may include convolutional neuralnetwork training. During preprocessing, one or more features common tomost scenarios and users, (“anchor features”), in a received image maybe detected. The detection may be made based on edge detection, shapedetection, and/or similar detection mechanisms, as are known. Based onthe location of the anchor feature(s), the images may be scaled androtated to make the image substantially level and with the anchorfeature(s) arranged in a predetermined position in the final image. Bydoing this, the training images may be consistently aligned, thusproviding more consistent results. The image may then be cropped to apredetermined area of pixels as input for further processing.

During preprocessing, data augmentation may also be performed to createadditional samples from the training images. For example, the inputimage may be randomly enlarged and/or shrunk, randomly rotated in aclockwise direction and/or in a counter clockwise direction, randomlycropped, and/or randomly changed with regard to saturation and/orexposure. In some instances, the input image may be subjected to randomvertical dropout, which randomly drops out a column of pixels (featuremap) of the image. The higher the layer, the more area of the elementthe dropout covers. By dropping out an entire column of pixels in theinput image, the CNN may learn to rely on a variety of features forsurface treatment evaluation, rather than one particular feature. Randomvertical dropout may also prevent over training of the CNN, thusmaintaining a desired accuracy level. Regardless of the techniquesimplemented, data augmentation allows the CNN to become more robustagainst variation in input images. This way, the CNN learns to extractimportant features under expected environmental variation caused by theway people take images, the conditions in which images are taken, andthe hardware used to take images.

Preprocessing may also include normalization. As an example, globalcontrast normalization may be utilized to standardize the trainingimages (and/or images of users). Similarly, the images may be maskedwith a fixed (or predetermined) size oval mask to minimize the influenceof other features. This also forces the CNN to learn and not depend ononly information in more fixed spatial location of the image.

During training, embodiments described herein may utilize mini-batchstochastic gradient descent (SGD) with Nesterov momentum (and/or otheralgorithms) to train the CNN. An example of utilizing a stochasticgradient descent is disclosed in US 8,582,807. The objective functionmay include a mean square error. In some embodiments, about 10% of thetraining subject may be withheld. The training error and validationerror on the withheld set may be monitored for training progress.

Once the CNN is trained, one or more of the CNN parameters may be fixed.As described in more detail below, a captured image may be forwardpropagated through the CNN to obtain a determined surface treatmentregimen, which can optionally be displayed to the user, for example, ona mobile computing device.

The CNN may include an inputted image, one or more convolution layersC1, C2, one or more subsampling layers S1 and S2, a fully integratedlayer, and an output. To begin an analysis or to train the CNN, an imageis inputted into the CNN (e.g., the image of a user). The CNN may sampleone or more portions of the image to create one or more feature maps ina first convolution layer C1. For example, the CNN may sample sixportions of the image to create six features maps in the firstconvolution layer C1. Next, the CNN may subsample one or more portionsof the feature map(s) in the first convolution layer C1 to create afirst subsampling layer S1. In some instances, the subsampled portion ofthe feature map may be half the area of the feature map. For example, ifa feature map comprises a sample area of 28 x 28 pixels from the image,the subsampled area may be 14 x 14 pixels. The CNN may perform one ormore additional levels of sampling and subsampling to provide a secondconvolution layer C2 and a second subsampling layer S2. It is to beappreciated that the CNN may include any number of convolution layersand subsampling layers as desired. Upon completion of final subsamplinglayer, the CNN generates a fully connected layer F1, in which everyneuron is connected to every other neuron. From the fully connectedlayer F1, the CNN can generate an output such as a predicted age or aheat map.

In some instances, at least some of the images and other data describedherein may be stored as historical data for later use. As an example,tracking of user progress may be determined based on this historicaldata. Other analyses may also be performed on this historical data,depending on the embodiment.

In one embodiment, a CNN based model is used for detecting and trackinggrooming implements 11 in a consumer video. The model utilizes multipleCNNs and other neural network components (such as a fully-connectednetwork, or an RNN) to accomplish this task. The image data 23 in theform of consumer video is fed into the model as a series of imageframes. Each image frame is first processed by a CNN to extract a set offeature maps (high-level features of the image). A second CNN, a RegionProposal Network, is used to propose a series of possible regions withinthe feature maps that might contain the grooming implement. The featuremaps within the proposed regions are then extracted to be furtherproposed to a fully connected network to decide whether the proposedregions contain a grooming implement, refine the proposed regionspositions, and map the coordinates of the proposed regions to theoriginal image. The end result is that for each image frame, the modelis able to decide whether a grooming implement exists, and if yes, theposition of the grooming implement within the image. Concurrently, theconsumer’s face can also be located using various facial recognitionalgorithms including CNN or any other facial detector algorithm. It ispossible to also have the face as part of the object detected in thedescribed region proposal network. It is also possible to overlay arecurrent neural network to capture temporal information of the video.By combining location information of both grooming implement andconsumer’s face, the implement can respond accordingly to provide bestgrooming experience. In one embodiment, the operating parameters of theimplement may be altered according to the manner in which the user isshaving, or otherwise grooming themselves or others. In one embodiment,the system may provide information to the user relating to the currentas well as historic uses of the grooming implement and the targetsurface.

The image(s), as well as one or more outputs from the neural network maybe passed to a database and aggregated with similar data from otherusers of the method. The aggregated data may be evaluated andcategorized into clusters using known clustering methods. The instantuser, surface and implement may then be associated with one or more ofthe now defined clustered populations based upon the data for the user,surface and implement. The association with particular clusterpopulations may then lead to the provision of cluster specificinformation to the user as part of the method. As an example, a user maybe categorized according to age, ethnicity and gender and a comparisonof the user’s data with that of the cluster population for the samegender, age and ethnicity may provide insights of use to thepractitioner when they are provided as part of the process.

In one embodiment, a method for treating a target surface 12 includesthe steps of: automatically evaluating digital image data 23 of thetarget surface 12. The digital image data 23 (which may be a collectionof images) may be provided to a machine learning classifier forevaluation. The collection of images may further include additional dataassociated with the content or context of the images. Data includingaudio, temperature, humidity or other environmental data capturedcontemporaneously with the images may also be provided to theclassifier. The classifier may previously have been trained to identifythe nature of the target surface 12 by presenting the classifier withtraining data including images of representative target surfaces eitherin isolation or together with other data as indicated above. The natureof the target surface 12 may include attributes including acategorization of the surface, such as skin, facial skin, teeth,fabrics, leather, plastics, wood, glass, ceramic, stone, or other hardor soft surfaces, as well as the current condition of the surface 12,the presence of facial hair, plaque, dirt, stains, and combinationsthereof upon the target surface may be determined by the analysis of thesurface via the images including the surface. The surface roughness orsurface finish may also be determined.

The analysis of the at least one image may further identify or determineat least one available treatment implement. In one embodiment, thedetermination may comprise determining the presence of a hand-heldtreatment implement. In one embodiment, the determination may be made bymatching content in the current images with images of suitableimplements presented in the training data set. In one embodiment, thedetermination may be made by inference wherein particular implements areassociated with certain surfaces according to previous definitions madeavailable for the analysis. In this embodiment, a toothbrush may beassociated with teeth, razors and implements with skin - body hair,scrub brushes with hard surfaces and so forth.

Further analysis of the at least one image and additional data maydetermine at least one surface treatment associated with the identifiedtarget surface in isolation or in conjunction with the identifiedtreatment implement. This determination may be made utilizing thedetermination of the nature of the surface, the treatment implement, thepractitioner, or combinations thereof. As an example, a grooming regimenmay be determined as appropriate for a combination of a groomingimplement and skin with hair. The analysis may then determine a use ofthe identified implement in the completion of the identified surfacetreatment.

Subsequent to the determinations, information analogous to thedetermined use may be provided to a user via a display system. Suchinformation may include specific instructions regarding the handling anduse of the implement in undertaking the treatment, the likely results ofundertaking the treatment, the progression of the treatment as evaluatedby the method over a series of treatments, the condition of theimplement relative to performing the treatment and so forth. Theinformation may be provided via digital display screen(s), by auditorycues from the implement or from distinct loudspeakers, or by visual cuessuch as indicator lights or other lighting changes in the treatmentenvironment. In one embodiment, the step of providing informationcomprises providing cues analogous to the spatial interaction betweenthe determined implement and the determined surface. In one embodiment,the step of providing information comprises providing informationanalogous to the temporal interaction of the determined implement andthe determined surface. In one embodiment, the step of providinginformation comprises providing information through an alteration of aproperty of the determined implement.

In one embodiment, the information to be presented may be stored in adatabase and called in response to the output of the CNN. The presentedinformation may be real-time information gathered during the treatment,information from the database and hybrid combinations of the two. As anexample, a template display of upper and lower teeth may be presented tothe user overlaid with real-time data illustrating which portions of theuser’s teeth have and have not been brushed during the current session.Data from the database illustrating trends of the user’s brushinghistory may be presented.

In one embodiment, the step of providing information may includeproviding information associated with the determined use, product orimplement as well as information associated with the user’s socialnetwork. Social network information accessed using account informationprovided by the user may enable the presentation of informationregarding similar treatments undertaken by members of the user’s socialnetwork including similarities and differences between the treatmentundertaken by the user and those treatments undertaken by other membersof the user’s social network. The social network information may also beused as an indicator of which social influencers are most likely to havean effect upon the user. This information may be used to selectcelebrity or social influencer how-to instructional content to presentto the user as well as product review and testimonial information fromthe identified influencers or nearest analog to the identifiedinfluencers.

In one embodiment, the method further comprises the step of providinginformation regarding a treatment implement relevant to the determinedsurface or surface treatment, wherein the implement is not detected inthe analysis of the data. As an example, analysis of the data mayindicate the use of a grooming implement without the use of acomplementary product which would improve or enhance the treatmentactivity. In this example, information regarding the missing product maybe provided to the user.

In one embodiment, the step of providing information may include agamification aspect. The information to be provided may be presented inthe form of a game for the user. The game may include aspects such aspoint scoring, competition with others, and rules of play. As anexample, use of an oral care implement such as a toothbrush may involvethe presentation of the information related to the time spent brushingand the areas of the oral cavity I, including dental surfaces, tongueand gums, treated thus far as well as remaining to be treated, may bepresented in the form of a game wherein the user must move the implementin a manner to clear objects from the display as a timer counts up ordown. In this embodiment, the graphic elements presented for removal maycoincide with the surfaces to be cleansed and may be removed from thedisplay only after sufficient time has been spent by the user intreating/cleaning those surfaces.

In one embodiment, the method may further comprise the step ofdetermining one or more properties of a treatment practitioner accordingto the evaluation of the at least one image. Properties including thepractitioner gender, dominant hand, skin condition, beard condition, maybe determined by analyzing the data and the context of the data togetherwith other information regarding the user and the environment of theuse. The determined properties of the practitioner may be used as inputsin determining what information to provide as the treatment activity isevaluated. Information specifically applicable to the user’s gender,dominant hand, skin condition, beard condition, and combinations thereofmay be provided.

In one embodiment, the information about the user may be combined withinformation about the product including brand, package quantity andquantity used for each treatment, to calculate product quantityremaining and thereby provide the user with an indication of when thecurrent product is likely to run out as well as an indication of whenthe product should be replaced or re-ordered using the user’s typicalmeans of acquiring the product.

In one embodiment, the method further comprises the step of determiningone or more environmental properties according to the evaluation of theone or more images together with at least one additional data source. Asan example, the method may determine the location of the practitionerand the surface, the time of day, the lighting available at the locationand other features of the local or external environment. The determinedenvironmental properties may be used as input in determining whatinformation to provide to the user as part of the method.

In one embodiment, the method may further comprise steps of: tracking aninitial determined treatment of a determined target surface; providinginformation analogous to the determined treatment, tracking andevaluating subsequent treatments of the target surface; and alteringsubsequent provided information according to a machine learningevaluation of the tracked initial and subsequent determined treatmentsand previously provided information. As an example, a user may use themethod to evaluate their shaving experience. Information may be providedto the use to enhance their shaving experience. Subsequent evaluationsmay indicate that portions of the previously provided information havebeen successfully followed or included in the shaving activity whileother portions have not yet been added successfully. Subsequent to thisdetermination, the provided information may be tailored to include onlythat information related to the portions which have not yet successfullybeen added to the treatment activity - in this example, shaving.Information types include shaving or treatment trends, ongoing treatmentresults - how well the user is shaving, what opportunities remain toimprove their experience, and diagnostic information relating to theuser’s grooming implement as well as their shaving activities.

In one embodiment, the method may further comprise steps of: tracking aninitial determined treatment of a determined target surface; tracking atleast one subsequent determined treatment of the same determinedtreatment surface; using machine learning in evaluating the combinationof tracked determined treatments of the determined target surface; andproviding information analogous to the determined treatment of thedetermined target surface according to the evaluation of the combinationof tracked determined treatments. The information provided may includeindications of improvements to the grooming activities as well asoutstanding opportunities for further improvements based upon aprogression in the grooming results.

In this embodiment, the step of: machine learning in evaluating thecombination of tracked determined treatments of the determined targetsurface, may comprise evaluating the practitioner in the combinationusing the environmental context of the treatment together with anyinformation provided by the user.

In this embodiment, the step of: machine learning in evaluating thecombination of tracked determined treatments of the determined targetsurface, may comprise evaluating the implement in the combination, theimplement may be evaluated in terms of the manufacturer and model of theimplement as well as the operational condition of the implementconsidered in terms of the implement’s performance in completing thesurface treatment. As an example, as the operating condition of theimplement declines, the work necessary to complete the task will change.

In this embodiment, the step of: machine learning in evaluating thecombination of tracked determined treatments of the determined targetsurface, may comprise evaluating the surface in the combination. Thenature of the surface may be evaluated to provide an input to thedetermination of the information to be provided. Evaluation of a user’sface may indicate a light or heavy growth of hair leading to theprovision of different information dependent upon the facial hairpresent at the time of treatment.

In one embodiment, the method further comprises the step of altering aperformance characteristic of the implement. In this embodiment, thedriven frequency of the implement may be changed to alter theperformance or to provide an auditory cue to the practitioner regardingthe treatment of the surface using the implement.

A system for practicing the methods may include a network, which may beembodied as a wide area network (such as a mobile telephone network, apublic switched telephone network, a satellite network, the internet,etc.), a local area network (such as wireless-fidelity, Wi-Max, ZigBee™,Bluetooth™, etc), and/or other forms of networking capabilities. Coupledto the network are a computing device, a remote computing device, akiosk computing device, and a training computing device.

The computing device may be a mobile telephone, a tablet, a laptop, apersonal digital assistant, an instrumented or smart mirror, and/orother computing device configured for capturing, storing, and/ortransferring images such as digital photographs and video. Accordingly,the mobile computing device may include an image capture device such asa digital camera, including depth sensing cameras and/or may beconfigured to receive images from other devices. The mobile computingdevice may include a memory component, which stores image capture logicand interface logic. The memory component may include random accessmemory (such as SRAM, DRAM, etc.), read only memory (ROM), registers,and/or other forms of computing storage hardware.

Recent advances in a machine learning technique called deep learninghave resulted in breakthrough performance in the realm of neuralnetworks. Examples of deep learning neural networks includeconvolutional neural network (CNN) and recurrent neural network (RNN).

CNN utilize a plurality of layers inspired by the human visual cortex. ACNN consists of an input and an output layer, as well as multiple hiddenlayers. The hidden layers of a CNN typically consist of convolutionallayers, pooling layers, fully connected layers and normalization layers.They have applications in a wide range of image and video applications,such as image classification, object detection, localization,segmentation, etc.

The CNN may be trained using predefined features and/or auto-learnedfeatures in a process called supervised learning. Supervised learninggenerally means that the CNN is trained by analyzing examples of imagesin which image classification/localization/detection/segmentation, etc.have been pre-defined. Depending on the accuracy that is desired, thenumber of training images may vary from a few images to a continuousinput of images to provide for continuous training. Regardless, aftertraining, the CNN learns key features for performing image related task.

After the CNN is trained, it may be used to generate an imageclassification, localization, detection, and/or segmentation related tooperation of a personal grooming appliance. In some instances, the CNNcan learn to identify facial/oral features, to recognize and localizegrooming appliance devices, identify treatment area and treatmentoptions, and to evaluate treatment results.

A recurrent neural network (RNN) is a class of deep learning neuralnetwork where connections between nodes form a directed graph along asequence. This allows it to exhibit temporal dynamic behavior for a timesequence. RNNs use their internal state (memory) to process sequences ofinput data. This makes them applicable to tasks such as machinetranslation, speech recognition, video analysis, sound detection, andmotion tracking, etc.

The RNN may be trained, similarly to CNN, using supervised learning.Supervised learning for RNN generally means the RNN is trained byanalyzing examples of sequential data, such as text, speech, sounds,videos, sensor data streams in which translated words, meaning ofsounds, action of videos, and corresponding physical measurement havebeen pre-defined. Depending on the accuracy that is desired, the numberof training samples may vary from a few short snippets of data streamsto a continuous input of data stream to provide for continuous training.Regardless, after training, the RNN learns key features for performingtasks involving sequential data.

After the RNN is trained, it may be used to analyze the data stream fromcameras or physical sensors and provide additional information relatedto operation of a personal grooming appliance. In some instances, theRNN can learn to localize the grooming applicator, identify movementpattern of grooming applicator, and/or to assess the usage of theapplicator.

Multiple type of deep learning neural networks are often usedsimultaneously to augment each other in order to achieve higherperformance. In some instances, CNN and RNN can be used independently toanalyze same or different streams of data, and the outputs fromdifferent neural network are jointly considered to drive user feedback.In other instances, a hybrid neural network architecture can beadopted - one hybrid neural network consists of both CNN and RNNbranches or layers. In one type of hybrid network, intermediate resultsof CNN and RNN are jointly feed into additional layers of neural networkto produce final output. In other type of hybrid networks, the output ofone network (CNN, for example) is fed into additional layer of network(e.g. RNN) for further processing before final result is obtained.

In one embodiment, a CNN and RNN is used to analyze the image from anexternal camera and motion and pressure sensor from an electric shaver.The CNN first identifies option of treatment, e.g. recommended shavingtechnique based on facial area and natural of facial hair, the RNN thenprovides real-time tracking of brush motion to insure consumer followthe recommendations, and CNN is used in the end to provide post-shavingevaluation.

In another embodiment, a hybrid CNN/RNN model is used to provide highlyaccurate tooth localization during brushing. A toothbrush is equippedwith both intra-oral camera and motion sensor and feed the hybridnetwork both vides stream and motion sensor stream. CNN and RNNcomponents of the hybrid network analyze the video stream and motionsensor stream, respectively to provide intermediate results onlocalization of the brush head inside the mouth. The intermediatelocalization results are further processed by additional layers ofneural network to yield an augmented localization of the brush-head aspart of the feedback to user for better brushing results.

Consequently, as will be appreciated, the internal sensor data 17 ₁, 17₂, 17 ₃, ..., 17 _(n) is not necessarily limited to motion sensor data,and the corresponding classifications 15 ₁, 15 ₂, 15 ₃, ..., 15 _(n) arenot necessarily limited to motion patterns. For example, the internalsensor data 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) may comprise data from one ormore pressure sensors, load sensors, temperature sensors, audiosensors/receivers, battery usage sensors, humidity sensors, bio-sensorsand the like (such internal sensors of the appliance may also bereferred to as “physical sensors”). Likewise, correspondingclassifications 15 ₁, 15 ₂, 15 ₃, ..., 15 _(n) may be the result ofclassifying data from one or more of the sensors, or combinations of twoor more of the sensors. Further, as will be discussed in further detailbelow, it is within the scope of the disclosure that the classifications15 ₁, 15 ₂, 15 ₃, ..., 15 _(n) may comprise hybrid or augmentedclassifications based upon a classification of a combination of internalsensor data 17 ₁, 17 ₂, 17 ₃, ..., 17 _(n) and image data 22.

FIG. 15 provides a schematic block diagram representation of anexemplary system according to an embodiment of the current disclosureutilizing such hybrid or augmented classifications. As shown in FIG. 15, a camera 1082/1070 is provided, which produces image data 1502 that isfed to a classifier 1504. Likewise, one or more sensors 1506 providesensor data 1508 that are fed to the same classifier 1504. Theclassifier 1504 has access to a knowledge base 1510 that has beentrained (and/or is being trained) based upon the various machinelearning techniques described herein. The trained knowledge base 1510 isaccessed by the classifier 1504 to classify the combination of imagedata 1502 and sensor data 1508 to provide a hybrid classification 1512that is accessible by the software application 1073. The classifier 1504may be any type of machine learning classifier and may utilize theneural networks (such as CNN and/or RNN) described herein, and theknowledge base 1510 comprises a trained (or training) set of potentialclassifications 15 ₁, 15 ₂, 15 ₃, ..., 15 _(n) and/or class members101A, 101B, 102A, 102B, ..., nA, nB for matching/classifying against theinput to the classifier(s).

For example, the classifier 1504 may be used to analyze an image from acamera 1070 associated with a smart phone or a smart mirror and thesensor data 1508 may include a pressure sensor data from an electricshaver. This hybrid classification 1512 produced by the classifier 1504may provide a combination of position and pressure information for useby the application 1073 for real time tracking of shaver motion andpressure to ensure, for example, a customer is following a recommendedshaving/treatment program. The classifier 1504, receiving the image data1502 and further having received the combination of image data andsensor data 1502 and 1508 may also provide a post-shaving evaluationbased upon the tracking classifications in combination with thepost-shaving image classification. As another example, the classifier1504 may be used to provide a highly accurate tooth localization duringbrushing. The toothbrush may be equipped with both an intra-oral camera1082 and motion sensors 1506 and the classifier 1504 may receive thecombination of the camera data 1502 and motion sensor data 1508 toprovide a hybrid classification 1512, where the hybrid classificationmay use the combination of motion sensor and intra oral imageinformation to provide an accurate localization of the brush headposition inside the mouth. The software application 1073 may utilizethis hybrid classification to generate feedback to the user for betterbrushing results.

This feedback may be provided, for example, to a display device1064/1066, to a sound actuator device 1067 and/or to one or more LEDs1050. The software application 1073 may also utilize the hybridclassification 1512 to adjust or modify the operation of the appliance.For example, the software application 1073 may adjust the operation of amotor 1514 present in the grooming appliance. As described herein,modifying the operation of the motor 1514 can be used, for example, tochange the speed of an electronic toothbrush operation, to change thespeed of an electronic shaving appliance operation, to change the angleof attack on a razor cartridge, to modify the speed or frequency ofoperation of a motor that controls the spinning or vibration of a brushor similar component of a household appliance and so forth. Likewise,the software application may utilize the hybrid classificationinformation 1512 to change various settings 1516 for operating thegrooming appliance and/or for operating the software application 1073.For example, depending upon the hybrid classification information 1512,device warning or notification settings may be altered (e.g., anover-pressure warning setting may be set at a different pressuredepending upon the location of a shaving appliance with respect to auser’s face or body-part).

As also shown in FIG. 15 , the software application can utilize thehybrid classification information 1512 as part of a training process forfurther training the knowledge base 1510. For example, based upon how auser interacts with the software application 1073, that user interactioninformation 1518 can be used by a training process 1520 to further trainthe knowledge base.

FIG. 16 provides a schematic block diagram representation of anexemplary system according to an embodiment of the current disclosureutilizing hybrid or augmented classifications generated in a differentmanner. Referring now to FIG. 16 , a schematic block diagramrepresentation of an alternate system is provided. In this examplesystem, the sensor data 1508 from the one or more sensors 1506 is fed toa sensor classifier 1602, while the image data 1502 received from thecamera 1082/1070 is fed to a separate image classifier 1604. The sensorclassifier classifies the sensor data based upon access to a trainedknowledge base 1606 while the image classifier classifies the image data1502 by accessing a train knowledge base 1608. The classifiers 1602/1604may be any type of machine learning classifier and may utilize theneural networks (such as CNN and/or RNN) described herein, and theknowledge bases 1606/1608 comprise trained (or training) sets ofpotential classifications (such as potential classifications 15 ₁, 15 ₂,15 ₃, ..., 15 _(n) and/or class members 101A, 101B, 102A, 102B, ..., nA,nB) for matching/classifying against the input to the classifier(s)1602/1604.

The sensor classifier 1602 generates one or more sensorclassification(s) 1610 that are fed to a reconciler 1612, while theimage classifier 1604 generates one or more image classification(s) 1614that are also sent to the reconciler 1612. The reconciler 1612 receivesthe sensor classification 1610 and image classification 1614 (and, insome embodiments, associated confidence values) and generates a hybridclassification 1616 based upon a combination of the sensorclassification(s) 1610 and image classification(s) 1614. The reconciler1612 may utilize any of the neural network (such as CNN and/or RNN)classifications described herein or may utilized other forms ofclassification, such as utilizing a form of statistical classification,such as multinomial logistic regression, methods (or alternateclassification methods) as known to those of ordinary skill. This hybridclassification 1616 information is then accessible by the softwareapplication 1073 for operation as described herein. Likewise, theknowledge base 1606 may be further trained by a training module 1618while knowledge base 1608 may be further trained by a training module1620. These training modules 1618/1620 may further train the respectiveknowledge bases 1606/1608 based upon user interaction informationreceived from the software application 1073.

The reconciler 1612 may be a separate module or may be incorporated intothe software application 1073. Additionally, the reconciler may not bepresent in all embodiments. For example, the sensor classification 1610and image classification 1614 may be separately provided to the softwareapplication 1073 where the software application 1073 may not necessarilygenerate a hybrid classification 1616 based upon a combination of thesensor classification and image classification. For example, asdescribed in the various use cases discussed herein, the softwareapplication 1073 may utilize an image classification initially toidentify a surface condition of a user’s body part (such as detectingthe presence of plaque on a user’s teeth or detecting whiskers on auser’s face or legs) and then developing a treatment schedule based uponthe image classification, where the treatment schedule may be providedto the user via the software application 1073. Subsequently followingthe development of the treatment program (such as recommendations as tohow to brush your teeth, how to apply cosmetics, or how to shave), thesoftware application may utilize the sensor classification information1610 to follow the user’s progress of treating the surface condition(such as brushing teeth, applying cosmetics, shaving a face or anotherbody part). The software application 1073 can then be used tocommunicate to the user progress information based upon the sensorclassification received.

It is also within the scope of the current disclosure that the treatmentplan can be based solely upon the image classification 1614 while theprogress information can be based upon the hybrid classification 1616.For example, the image classification 1614 may be used to determine asurface condition and, based upon the surface condition, the softwareapplication may establish a treatment plan. Thereafter, the softwareapplication may use the hybrid classification 1616 (which utilizes acombination of subsequent sensor classification(s) 1610 and subsequentimage classification(s) 1614) to follow the progress of the treatment ofthe surface condition. Thereafter, based upon how the user isprogressing with respect to the treatment plan established initially,the software application 1073 may communicate to the user how the useris performing with respect to the treatment plan, may modify thetreatment plan or may correct the initial image classification(indicating that the initial image classification may have detected theincorrect surface condition) and develop a new treatment plan based uponthe revised indication.

The example system shown in FIGS. 15 and 16 can be used for many usecases including, but not limited to grooming appliance and householdappliance use cases. Only a number of potential use cases will bediscussed herein, but it will be appreciated that many more areenvisioned and within the scope of the current disclosure.

In a first use case example, the image data 1502 may be utilized toindicate the lubrication product (i.e., shaving gel) being used (such asby brand or type); and may also indicate the facial hair area beingshaved, the direction of the shave, and/or the delay time betweenapplying the shaving lubricant and the act of shaving. In the sameexample, the sensor data 1508 may be MEMS (microelectromechanicalsystems) motion information, speed information, pressure information ofa razor on the face, and/or positional information. The hybridclassification 1512/1616, in this example, may be used by the softwareapplication 1073 for various benefits. For example, as razors wear down,efficacy decreases and the consumer typically speeds up strokes andapplies additional pressure to compensate. The software application 1073can detect subtle changes in the user’s routine (shave metrics) andchanges in the components of the shave (such as the shave gel type) torecommend to the user changes in the routine and/or components to ensurea successful shaving experience. Alternatively, the software applicationcan use the hybrid classification 1512/1616 to detect a need for adifferent angle of attack of the shaving cartridge or a different levelof resistance in the pivoting of the shaving cartridge and modifyoperation of a component 1514 and/or modify a setting 1516 of theappliance accordingly.

In another exemplary use case, the image data 1502 may be an image of auser’s face, while the sensor data 1508 may include MEMS sensorinformation pertaining to a location of the grooming appliance withrespect to the user’s face. The software application 1073 may analyzethe image classification 1614 or the hybrid classification 1512/1616 todetermine an emotional response of the user while using a groomingappliance (such as a shaving appliance, dental appliance or cosmeticapplicator); and, likewise, use the positional information present inthe sensor classification 1610 or hybrid classification 1512/1616 todetermine where the grooming appliance was located (and/or how thegrooming appliance was being used) at the time of the experience of theemotional reaction. The application 1073 may then use that combinationof information to provide feedback to the user or to modify operation ofa component 1514 and/or modify a setting 1516 of the appliance. As anexample, if the user shows a negative emotion shaving his neck, and thehybrid classification 1512/1616 indicates that a certain pressure andshaving direction was being used at that time, the application 1073 maysuggest to the user a different way to shave his neck the next time.

In another example use case, the image data 1502 may be used to providepre- and/or post-shave information while the sensor data 1508 mayinclude location information, speed of movement information, and/orpressure information of a shaving device. Using this combination ofinformation as described above, the application 1073 can analyze theimages before shaving to determine direction of whiskers and a besttreatment approach to obtain a close shave. Thereafter, data from thesensor information 1508 and/or hybrid classification 1512/1616 mayindicate how the shave was performed as compared to the recommendations.The post-shaving image information may then be analyzed and theapplication may refine its guidance for a better shave the next time theuser attempts to use the shaving appliance. For example, if the postshave images show irritation and the guidance was followed, theapplication may offer post shave options (such as shave ball) orrecommend pre shave routines (such as using a warm towel, a new blade,and/or a different shaving foam) to help minimize issues. Theapplication 1073 may also tag the consumer for future productdevelopment follow-up (for example, if there is a tendency for theconsumer to get in-grown hairs after shaving, the application 1073 mayoffer marketing information to the user different products that may beutilized to avoid in-grown hairs in the future).

In a next example use case, the combination of image classification andsensor classifications may be utilized in a marketing context. Forexample, the image data 1502 may provide, during brushing, the productsused by the user (power brush vs. manual, toothbrush, toothpaste type,mouthwash, floss, etc.) while the sensor data 1508 may provide location,speed and pressure information during the brushing activity. Theapplication 1073, leveraging the hybrid classification 1512/1616 andcontextual data (such as the identity of the user, the age of the user,the ethnicity of the user, user’s habits, products present, etc.) maycross-sell consumers to different products. For example, a consumer whois brushing with a manual toothbrush and uses sensitivity toothpaste maybe receptive to messaging, coupons, samples of a power brush with softbristles and/or a different toothpaste used to minimize futuresensitivity issues.

In another use case, the image data 1502 may be used by the softwareapplication 1073 as classifying a skin age analysis before product use,and may also be utilized during use for locational information. Thesensor data 1508 may be various performance conditions of the beautyproduct applicator being used, such as speed of use, pressure and thelike. Then, based upon this combination of information, the application1073 may recommend a cosmetic and an application technique to the userthrough the application 1073 to maximize performance. The application1073 may leverage data from the sensor data from the applicator deviceto understand how (pressure and motion) the product is being applied,patted, rubbed, dotted). The application 1073 may track and coach ontechnique to encourage adhering to the guidance using the application.

In another use case example, a camera 1082 located on a leading end of adental appliance may be used to identify the presence and locations ofplaque on a user’s teeth. Based upon this identification of the locationand presence of plaque on the user’s teeth, the software application1073 may thereafter generate a treatment plan which will be communicatedto the user. This treatment plan may provide a recommendation for how tobrush the user’s teeth using the manual or electronic toothbrush and mayalso guide the user to the various tooth locations in real time duringbrushing (such as via a display 1066 on a smart mirror device). Forexample, while the user is brushing before a smart mirror device, theremay be an animated display on the corner of the device showing to theuser where the user has brushed and where the user has not brushed andwhere the presence of plaque may still be indicated so that the userwill know what parts of the user’s mouth to complete brushing beforefinishing the brushing activity. Consequently, during brushing, thehybrid classification 1616/1512 may be a hybrid combination ofpositional data from the sensor data 1508 and image data 1502 to provideto the application 1073 progress information of how the user is doingwith the brushing activity.

In a similar example, an external camera 1070 located on a computerizedsmart phone or smart mirror 1080 may be used to analyze a target such asa room, a floor, a window and the like and the application 1073 may beused to determine the state of the target. Using this information, thesoftware application 1073 may set goals or a treatment plan for usingthe grooming appliance or household appliance for treating the target.Thereafter, once implemented, the software application 1073 may utilizethe sensor information 1602 (which may include motion sensor data) orthe hybrid classification 1512/1616 to monitor the use of the appliancewith respect to the target to determine whether or not the goals are oftreating the state of the target are accomplished.

In another example, the image classifier 1604 may be used to determinethe identification of the target and the appliance being used withrespect to the target (a user’s face and a shaving appliance).Thereafter, the sensor classification 1610 or the hybrid classification1512/1616 may be used to determine how the grooming appliance isengaging with the target. The software application 1073 may utilize ahybrid classification 1512/1616 to obtain a refined determination of howthe implement and the target are interacting and based upon this hybridclassification information may provide user feedback in real-time orafter use.

As some additional examples when operating as a toothbrush, theapplication can utilize the hybrid classification 1512/1616 to determinewhen a toothbrush is outside of a user’s mouth and based upon thisdetermination may disable or turn off the device, or at least disable orturn off the motor 1514 operating the motorized toothbrush. As anotherexample, based upon this hybrid classification information 1512/1616 thesoftware application 1073 may change the color of a multi-color LED 1050(such as a smart ring) based upon the software application utilizing thehybrid classification 1512/1616 to determine the identity or othercharacteristic of the user. In another example, the hybridclassification 1512/1616 may be used to detect a brush head type of thegrooming appliance and then be used to determine whether or not tochange the speed setting 1516 of the brush (a detection of a softbristle may result in setting a default speed to “gentle”). As anotherexample, the hybrid classification 1512/1616 may be used to detect brushposition with respect to the user and then the software application 1073may automatically adjust the speed of the brush by modifying operationof the motor 1514 or by modifying a speed setting 1516 (for example,when the hybrid classification indicates that the brush is positioned ona tongue of the user, the operation of the device is modified based upontongue cleaning settings). As another example, the hybrid classification1512/1616 may be used to determine the location of a shaving appliancewith respect to a user’s skin and then adjust a pressure warning settingdepending upon the area of the grooming device with respect to the user(for example, if the grooming device is on a position of the skin wheresensitivity is not likely, the settings 1516 may be modified so that apressure warning produced by speaker 1066 will only be activated at ahigher level of pressure is sensed as compared to a lower level if thehybrid classification indicates that the shaving device is located in anarea of high sensitivity). As another example, the hybrid classification1512/1616 can be used to determine how long the grooming appliance hasbeen used since a grooming implement, such as a razor cartridge ortoothbrush head, has been changed. Consequently, the application 1073may use this information to advise the user that it’s time to change thebrush head of a toothbrush or a shaving cartridge on a shaver if theappliance has been used for a longer than a recommended replacementschedule.

As discussed above, the software application 1073 can use the hybridclassifications 1512/1616 to adjust the operation of the groomingappliance or household appliance in many different ways. For example,the software application 1073 may turn on or off the grooming applianceor implements contained in the appliance; may adjust the speed of theimplements of the grooming/household appliance; may adjust the pressuresettings permitted before warning of the grooming/household appliance;may activate lights, LEDs, colors, etc. based upon the operation of thegrooming appliance/household appliance; may adjust the operation of theappliance for maintenance related issues such as compensating for lackof maintenance (old shaving foils, old brushes, etc.); may providefeedback to the user suggesting replacement of an implement, such asrecommending replacing a worn out brush or shaving cartridge, etc.; mayadjust operational settings, such as angle of attack for a shavingdevice based on facial location; and may adjust the stiffness ormaneuverability of the appliance implements (such as the pivotingstiffness of a razor cartridge).

As discussed above, the application 1073 may provide various forms offeedback to the user of the appliance. Such feedback may be visual (suchas provided through a networked user interface device 1080 such as asmart phone, tablet, personal assistant, or smart mirror device), audio(spoken, sounds, etc) from the appliance itself or from some otherdevice such as a personal assistant device, haptic feedback from theappliance or from some other source and the like. Of course, thefeedback may be any combination of visual, audio and haptic. Forexample, the feedback may be in the form of an animated video presentedon the device 1080 before, during and/or after use of the appliance.

The training modules 1520/1618/1620 may train the classifiers based uponuser interaction with the application 1073 as discussed above. But thetraining may also be based upon an individual’s use of the appliance(e.g., left-handed vs. right-handed). The training may also be acrowd-based training where multiple users train the classifiers based onoverall habits and usage.

The application 1073 may also train the classifiers based upon how theuser follows (or fails to follow) treatment advice or other feedback.

Embodiments disclosed herein may use the augmented or hybridclassification 1512/1616 to determine the relative position/location ofa target surface and the nature of the target surface of a subject. Anembodiment provides an appliance, comprising a sensor equipped appliancefor the acquisition and transmission of the sensor data 1508 and forreceiving and processing of the sensor data to determine the relativeposition of a target surface and the nature of the target surface.

The assessment of a target surface including the acquisition andprocessing of at least one digital image 1502 of the target surface andits nature, consisting of information of oral conditions, issues anddiseases including, but not only limited to plaque, stain, calculus,discoloration, early and late stages of caries, white spot lesion,fluorosis, demineralization, gingivitis, bleeding, gingival recession,periodontitis, fistula, gingival abrasion, aphthous, other lesions ofmucosa and structure and cleanliness of tongue. The assessment of atarget surface includes also the determination of sound teeth, gums,mucosa and tongue, as well as the determination of missing teeth, teethalignment and artificial materials like dental implants, dentures,crowns, inlays, fillings, brackets and other tooth position correctingappliances. The assessment of a target surface can also be part of adetermination of an oral health index generated via intra-oral-camerasand smart analysis systems based on, but not limited to, machinelearning, deep learning and artificial intelligence.

The assessment results from position, target surface and its nature(data) analyzed over time help to drive the robustness of the analysisfor endpoints like plaque, gingivitis and other endpoints listed above.For the acquisition of data, the sensor equipped device may use sensorssuch as, but not limited to, optical sensors, cameras, bio-sensors andInertia Measurement Units. For the acquisition of data, the sensorequipped device may use additional light sources to allow to detectendpoints like, but not limited to, plaque and to optimize theenvironmental conditions for the sensors. The sensor equipped device mayuse preprocessing or filtering methods to alter the sensor data beforetransmission. For processing the acquired position, target surface andnature of target surface (data), the software within the system may usemathematical methods such as but not limited to statistical analysis,machine learning, deep learning, artificial intelligence, etc such asthose described herein.

In an embodiment, the device may contain software for operation of thedevice and for processing and displaying of the position, the targetsurface and the nature of the target surface (data). In an embodiment,the position, the target surface and the nature of the target surface isdisplayed either in real-time/live during the data acquisition or afterdata processing. In this setup, the target surface and the nature of thetarget surface data can be shown solitary or can be combined with theposition data to be projected to a real or abstract model.

With respect to the embodiments of FIGS. 15 and 16 , the bio-sensors maybe utilized in place of the cameras 1082/1070 so that the augmented orhybrid classifications 1512/1616 may be based upon a classification of acombination of bio-sensor data and internal sensor data 1508. In anexample, the system could detect gingivitis on the upper buccal dentalzone. This information is transferred to an Oral Care cleaning system sothat, when brushing in the specified zone, the application 1073 changesthe brushing mode setting 1516 to “sensitive” and the “pressurethreshold” settings 1516 will be set lower.

As another modified use case, a diagnosis of a medical conditioncombined with the location of that condition may also result in amodification of the operation of the appliance in that location. Forexample, an operation of an electric toothbrush may be automaticallyadjusted to a “sensitive” mode of operation when the hybridclassification 1512/1616 determines that the toothbrush is in a locationof the mouth where gingivitis has been (or is being) diagnosed.

It will be apparent that changes or modifications may be made to theexemplary embodiments without departing from the scope as claimed below.Furthermore, it is also not necessary that the any objects or advantagesdiscussed herein be present to fall within the scope since manyadvantages may be present without necessarily being disclosed herein.

The dimensions and values disclosed herein are not to be understood asbeing strictly limited to the exact numerical values recited. Instead,unless otherwise specified, each such dimension is intended to mean boththe recited value and a functionally equivalent range surrounding thatvalue. For example, a dimension disclosed as “40 mm” is intended to mean“about 40 mm”.

Every document cited herein, including any cross referenced or relatedpatent or patent publication, is hereby incorporated herein by referencein its entirety unless expressly excluded or otherwise limited. Thecitation of any document is not an admission that it is prior art withrespect to any document disclosed or claimed herein or that it alone, orin any combination with any other reference or references, teaches,suggests or discloses any such embodiment. Further, to the extent thatany meaning or definition of a term in this document conflicts with anymeaning or definition of the same term in a document incorporated byreference, the meaning or definition assigned to that term in thisdocument shall govern.

The dimensions and values disclosed herein are not to be understood asbeing strictly limited to the exact numerical values recited. Instead,unless otherwise specified, each such dimension is intended to mean boththe recited value and a functionally equivalent range surrounding thatvalue. For example, a dimension disclosed as “40 mm” is intended to mean“about 40 mm”.

What is claimed is:
 1. A method for treating a surface of a user’s bodypart, comprising: obtaining target surface condition information from auser’s body part surface using a bio-sensor that is not a camera;classifying the target surface condition information using a machinelearning classifier to determine an initial target surface conditionclassification; obtaining treatment progress information using acombination of motion sensor data and surface condition information fromthe bio-sensor, displaying a real time representation of the treatmentprogress information, and modifying setting of a treatment system basedupon, at least in part, the treatment progress information; andclassifying the treatment progress information using a machine learningclassifier to determine a progress classification for treating theinitial target surface condition classification.
 2. The method of claim1, wherein the bio-sensor is provided on at least one of an examinationinstrument or a grooming appliance.
 3. The method of claim 1, whereinthe modifying step occurs substantially in real time while the treatmentsystem is treating the user’s body part surface.
 4. The method of claim1, wherein the bio-sensor is provided a treatment instrument of thetreatment system.
 5. The method of claim 4, wherein the treatmentinstrument is an oral care instrument.
 6. The method of claim 1, furthercomprising modifying settings of a treatment system based upon thetarget surface condition classification.
 7. The method of claim 1,further comprising evaluating change of the user’s body part surfacecondition over time based upon successive target surface conditionclassifications.
 8. The method of claim 1, wherein the progressclassification indicates that the initial target surface conditionclassification is incorrect.
 9. The method of claim 8, furthercomprising generating an initial treatment plan based upon the initialtreatment classification and modifying the initial treatment plan upondetermining that the initial target surface condition classification isincorrect.